Hi, We are experiencing regular "freezes" of bird6 on a BGP router. When this happens, bird6 maxes out a CPU for several minutes. If a command is run in birdc6 during such a freeze, the command hangs, and the result is only returned when birdc6 has stopped using the CPU. Note that this also applies to "cheap" commands like "show protocols", which usually complete instantly (both with bird, and with bird6 in non-freeze conditions). Sometimes (but not always), the non-responsiveness of bird6 causes all BGP sessions to drop, which is really annoying on a full-view BGP router. The freezes happen at random, but seem to happen more frequently when the router is under load (typically, at peak time, each CPU spends ~20% forwarding packets, on a 4-core box). The BGP setup is made of multiple transit and peerings, on multiple VLANs (some BGP neighbours share the same VLAN). The setup is pretty similar on bird and bird6, but only bird6 exhibits these freezes, bird works just fine. The box is running Debian wheezy on amd64, with bird from backports: 1.4.5-1~bpo70+1 Attached is the configuration, and two extracts of the logs when all BGP sessions dropped (with debug { states, interfaces, events }). All files are anonymised, but should be consistent. What do you think? It looks like bird6 gets stuck on some very expensive operation, which prevents it from doing anything else (include maintaining BGP sessions alive). Thanks, Baptiste
I just tried downgrading from 1.4.5 to 1.4.4, using the 1.4.4-1~bpo70+1 Debian package from http://bird.network.cz/?download&tdir=debian/ The result is the same, bird6 also freezes periodically with version 1.4.4. By the way, I think I ruled out the possibility that a particular BGP peer is sending garbage: the issue still arises when leaving only one BGP session active, whichever it is. Is there anything else I can do to help troubleshoot the root cause of this issue? On Thu, Jan 29, 2015 at 08:03:07PM +0100, Baptiste Jonglez wrote:
Hi,
We are experiencing regular "freezes" of bird6 on a BGP router. When this happens, bird6 maxes out a CPU for several minutes. If a command is run in birdc6 during such a freeze, the command hangs, and the result is only returned when birdc6 has stopped using the CPU. Note that this also applies to "cheap" commands like "show protocols", which usually complete instantly (both with bird, and with bird6 in non-freeze conditions).
Sometimes (but not always), the non-responsiveness of bird6 causes all BGP sessions to drop, which is really annoying on a full-view BGP router.
The freezes happen at random, but seem to happen more frequently when the router is under load (typically, at peak time, each CPU spends ~20% forwarding packets, on a 4-core box).
The BGP setup is made of multiple transit and peerings, on multiple VLANs (some BGP neighbours share the same VLAN). The setup is pretty similar on bird and bird6, but only bird6 exhibits these freezes, bird works just fine.
The box is running Debian wheezy on amd64, with bird from backports: 1.4.5-1~bpo70+1
Attached is the configuration, and two extracts of the logs when all BGP sessions dropped (with debug { states, interfaces, events }). All files are anonymised, but should be consistent.
What do you think? It looks like bird6 gets stuck on some very expensive operation, which prevents it from doing anything else (include maintaining BGP sessions alive).
Thanks, Baptiste
If built with symbols, after it has gotten into the CPU busy-loop, use gdb to attach to it, ala: gdb <path to bird6> <process id of bird6> Ex: gdb /usr/local/sbin/bird6 `ps -C bird6 -o pid=` then "bt for a stack trace, possibly showing where stuck. "cont" to continue and then another control-c to check again. Do this a few times. Hopefully there will be a pattern. Copy & paste the results to this list. "quit" to exit gdb, allowing bird6 to continue. Chris On Sat, 31 Jan 2015, Baptiste Jonglez wrote:
I just tried downgrading from 1.4.5 to 1.4.4, using the 1.4.4-1~bpo70+1 Debian package from http://bird.network.cz/?download&tdir=debian/
The result is the same, bird6 also freezes periodically with version 1.4.4.
By the way, I think I ruled out the possibility that a particular BGP peer is sending garbage: the issue still arises when leaving only one BGP session active, whichever it is.
Is there anything else I can do to help troubleshoot the root cause of this issue?
On Thu, Jan 29, 2015 at 08:03:07PM +0100, Baptiste Jonglez wrote:
Hi,
We are experiencing regular "freezes" of bird6 on a BGP router. When this happens, bird6 maxes out a CPU for several minutes. If a command is run in birdc6 during such a freeze, the command hangs, and the result is only returned when birdc6 has stopped using the CPU. Note that this also applies to "cheap" commands like "show protocols", which usually complete instantly (both with bird, and with bird6 in non-freeze conditions).
Sometimes (but not always), the non-responsiveness of bird6 causes all BGP sessions to drop, which is really annoying on a full-view BGP router.
The freezes happen at random, but seem to happen more frequently when the router is under load (typically, at peak time, each CPU spends ~20% forwarding packets, on a 4-core box).
The BGP setup is made of multiple transit and peerings, on multiple VLANs (some BGP neighbours share the same VLAN). The setup is pretty similar on bird and bird6, but only bird6 exhibits these freezes, bird works just fine.
The box is running Debian wheezy on amd64, with bird from backports: 1.4.5-1~bpo70+1
Attached is the configuration, and two extracts of the logs when all BGP sessions dropped (with debug { states, interfaces, events }). All files are anonymised, but should be consistent.
What do you think? It looks like bird6 gets stuck on some very expensive operation, which prevents it from doing anything else (include maintaining BGP sessions alive).
Thanks, Baptiste
if the pid of bird6 is 123, run (as root) strace -p 123 -o /tmp/bird6.strace.out and then send in the bird6.strace.out. strace uses the same hooks gdb uses but doesnt stop the process. The outfile will contain all the system calls and their args from the process in real time as it is running. control-C will kill strace and which should cause it to detach and let bird6 keep going unfettered. Ed -----Original Message----- From: bird-users-bounces@network.cz [mailto:bird-users-bounces@network.cz] On Behalf Of Chris Caputo Sent: Friday, January 30, 2015 7:28 PM To: Baptiste Jonglez Cc: bird-users@network.cz Subject: Re: Bird6 freeze under high load If built with symbols, after it has gotten into the CPU busy-loop, use gdb to attach to it, ala: gdb <path to bird6> <process id of bird6> Ex: gdb /usr/local/sbin/bird6 `ps -C bird6 -o pid=` then "bt for a stack trace, possibly showing where stuck. "cont" to continue and then another control-c to check again. Do this a few times. Hopefully there will be a pattern. Copy & paste the results to this list. "quit" to exit gdb, allowing bird6 to continue. Chris On Sat, 31 Jan 2015, Baptiste Jonglez wrote:
I just tried downgrading from 1.4.5 to 1.4.4, using the 1.4.4-1~bpo70+1 Debian package from http://bird.network.cz/?download&tdir=debian/
The result is the same, bird6 also freezes periodically with version 1.4.4.
By the way, I think I ruled out the possibility that a particular BGP peer is sending garbage: the issue still arises when leaving only one BGP session active, whichever it is.
Is there anything else I can do to help troubleshoot the root cause of this issue?
On Thu, Jan 29, 2015 at 08:03:07PM +0100, Baptiste Jonglez wrote:
Hi,
We are experiencing regular "freezes" of bird6 on a BGP router. When this happens, bird6 maxes out a CPU for several minutes. If a command is run in birdc6 during such a freeze, the command hangs, and the result is only returned when birdc6 has stopped using the CPU. Note that this also applies to "cheap" commands like "show protocols", which usually complete instantly (both with bird, and with bird6 in non-freeze conditions).
Sometimes (but not always), the non-responsiveness of bird6 causes all BGP sessions to drop, which is really annoying on a full-view BGP router.
The freezes happen at random, but seem to happen more frequently when the router is under load (typically, at peak time, each CPU spends ~20% forwarding packets, on a 4-core box).
The BGP setup is made of multiple transit and peerings, on multiple VLANs (some BGP neighbours share the same VLAN). The setup is pretty similar on bird and bird6, but only bird6 exhibits these freezes, bird works just fine.
The box is running Debian wheezy on amd64, with bird from backports: 1.4.5-1~bpo70+1
Attached is the configuration, and two extracts of the logs when all BGP sessions dropped (with debug { states, interfaces, events }). All files are anonymised, but should be consistent.
What do you think? It looks like bird6 gets stuck on some very expensive operation, which prevents it from doing anything else (include maintaining BGP sessions alive).
Thanks, Baptiste
On Sat, Jan 31, 2015 at 12:37:09AM +0000, Pendzik, Edward wrote:
if the pid of bird6 is 123, run (as root)
strace -p 123 -o /tmp/bird6.strace.out
and then send in the bird6.strace.out.
Well, i would guess that it is unlikely that such lockup would do some system calls. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hi, Thanks for this strace trick, I didn't know one could attach to a running process like this. The information returned by strace is not really helpful, since it consists in a massive amount of netlink messages (and strace doesn't decode netlink messages). However, it indicates that Bird spends a lot of time communicating with the kernel, which is a valuable information in itself :) After activating the kernel logs in bird (debug { packets }), here is a sample of the logs during a freeze: Jan 31 12:06:26 myrouter bird6: kernel1: 2001::/32: seen Jan 31 12:06:26 myrouter bird6: kernel1: 2001::/32: already seen Jan 31 12:06:32 myrouter bird6: kernel1: 2001::/32: already seen Jan 31 12:06:32 myrouter bird6: kernel1: 2001::/32: already seen Jan 31 12:06:38 myrouter bird6: kernel1: 2001::/32: already seen Jan 31 12:06:50 myrouter bird6: kernel1: 2001::/32: already seen Jan 31 12:06:50 myrouter bird6: kernel1: 2001::/32: already seen Jan 31 12:06:50 myrouter bird6: kernel1: 2001::/32: already seen Jan 31 12:06:56 myrouter bird6: kernel1: 2001::/32: already seen So, it looks like bird is receiving many times the same routes from the kernel when it scans its routing table. It seems related to either netlink or the kernel, since "ip" also exhibits the same behaviour: $ ip -6 r | wc -l 7690163 This took several minutes to complete, and there certainly isn't so much IPv6 routes in the kernel: routes appear several times in the output of "ip -6 r". Running this command multiple times yields very different results each time. Thus, I don't think the bug is in Bird. Could it be some kind of race condition with netlink? I haven't been able to find any reference to this bug, either in the kernel or in iproute2. For reference, this is on a Debian wheezy system, but I can reproduce the duplicate routes in "ip -6 r" on Debian jessie as well. Thanks, Baptiste PS: as a temporary solution, I increased the scan time of the kernel protocol from one minute to several hours, so that the issue arises less often. On Sat, Jan 31, 2015 at 12:37:09AM +0000, Pendzik, Edward wrote:
if the pid of bird6 is 123, run (as root)
strace -p 123 -o /tmp/bird6.strace.out
and then send in the bird6.strace.out.
strace uses the same hooks gdb uses but doesnt stop the process. The outfile will contain all the system calls and their args from the process in real time as it is running. control-C will kill strace and which should cause it to detach and let bird6 keep going unfettered.
Ed
-----Original Message----- From: bird-users-bounces@network.cz [mailto:bird-users-bounces@network.cz] On Behalf Of Chris Caputo Sent: Friday, January 30, 2015 7:28 PM To: Baptiste Jonglez Cc: bird-users@network.cz Subject: Re: Bird6 freeze under high load
If built with symbols, after it has gotten into the CPU busy-loop, use gdb to attach to it, ala:
gdb <path to bird6> <process id of bird6>
Ex:
gdb /usr/local/sbin/bird6 `ps -C bird6 -o pid=`
then "bt for a stack trace, possibly showing where stuck.
"cont" to continue and then another control-c to check again.
Do this a few times. Hopefully there will be a pattern.
Copy & paste the results to this list.
"quit" to exit gdb, allowing bird6 to continue.
Chris
On Sat, 31 Jan 2015, Baptiste Jonglez wrote:
I just tried downgrading from 1.4.5 to 1.4.4, using the 1.4.4-1~bpo70+1 Debian package from http://bird.network.cz/?download&tdir=debian/
The result is the same, bird6 also freezes periodically with version 1.4.4.
By the way, I think I ruled out the possibility that a particular BGP peer is sending garbage: the issue still arises when leaving only one BGP session active, whichever it is.
Is there anything else I can do to help troubleshoot the root cause of this issue?
On Thu, Jan 29, 2015 at 08:03:07PM +0100, Baptiste Jonglez wrote:
Hi,
We are experiencing regular "freezes" of bird6 on a BGP router. When this happens, bird6 maxes out a CPU for several minutes. If a command is run in birdc6 during such a freeze, the command hangs, and the result is only returned when birdc6 has stopped using the CPU. Note that this also applies to "cheap" commands like "show protocols", which usually complete instantly (both with bird, and with bird6 in non-freeze conditions).
Sometimes (but not always), the non-responsiveness of bird6 causes all BGP sessions to drop, which is really annoying on a full-view BGP router.
The freezes happen at random, but seem to happen more frequently when the router is under load (typically, at peak time, each CPU spends ~20% forwarding packets, on a 4-core box).
The BGP setup is made of multiple transit and peerings, on multiple VLANs (some BGP neighbours share the same VLAN). The setup is pretty similar on bird and bird6, but only bird6 exhibits these freezes, bird works just fine.
The box is running Debian wheezy on amd64, with bird from backports: 1.4.5-1~bpo70+1
Attached is the configuration, and two extracts of the logs when all BGP sessions dropped (with debug { states, interfaces, events }). All files are anonymised, but should be consistent.
What do you think? It looks like bird6 gets stuck on some very expensive operation, which prevents it from doing anything else (include maintaining BGP sessions alive).
Thanks, Baptiste
On Sat, Jan 31, 2015 at 02:47:51PM +0100, Baptiste Jonglez wrote:
This took several minutes to complete, and there certainly isn't so much IPv6 routes in the kernel: routes appear several times in the output of "ip -6 r". Running this command multiple times yields very different results each time.
Thus, I don't think the bug is in Bird. Could it be some kind of race condition with netlink? I haven't been able to find any reference to this bug, either in the kernel or in iproute2. For reference, this is on a Debian wheezy system, but I can reproduce the duplicate routes in "ip -6 r" on Debian jessie as well.
Interesting, what are the kernel versions in these Wheezy and Jessie systems? Does the problem (with 'ip -6 r') appears also when BIRD is not running? I wonder what factors are specific to this problem. I remember there were a similar report or two few years ago, but these reports are too uncommon to be an universal problem in IPv6 Linux forwarding. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Sat, Jan 31, 2015 at 03:06:37PM +0100, Ondrej Zajicek wrote:
On Sat, Jan 31, 2015 at 02:47:51PM +0100, Baptiste Jonglez wrote:
This took several minutes to complete, and there certainly isn't so much IPv6 routes in the kernel: routes appear several times in the output of "ip -6 r". Running this command multiple times yields very different results each time.
Thus, I don't think the bug is in Bird. Could it be some kind of race condition with netlink? I haven't been able to find any reference to this bug, either in the kernel or in iproute2. For reference, this is on a Debian wheezy system, but I can reproduce the duplicate routes in "ip -6 r" on Debian jessie as well.
Interesting, what are the kernel versions in these Wheezy and Jessie systems? Does the problem (with 'ip -6 r') appears also when BIRD is not running?
So far, I saw this behaviour in Wheezy (3.2.0-4-amd64 + iproute 20120521-3+b3), and Jessie (3.14.4-1 + iproute2 3.16.0-2). In both cases, Bird was running. I tried shutting down bird6 (with "persist" in the kernel protocol, so that routes stay in the kernel), and the problem seemed to persist: $ ip -6 r | wc -l 26720 $ ip -6 r | wc -l 21794 $ ip -6 r | wc -l 50321 $ ip -6 r | wc -l 37602 $ ip -6 r | wc -l 59011 However, the amplitude is very reduced (when Bird is running and talking to the kernel, doing the same thing yields millions of routes).
I wonder what factors are specific to this problem. I remember there were a similar report or two few years ago, but these reports are too uncommon to be an universal problem in IPv6 Linux forwarding.
I actually found a way to reproduce this without Bird, just by inserting static routes. It seems that dumping the routing table while some routes are being inserted is enough to trigger the bug. Simple way to see the issue: run this on one shell watch -n 0.2 "ip -6 r | wc -l" and run this in another, root, shell: for i in {0000..3999}; do ip -6 r add unreachable 2001:db8:$i::/48 proto 57; sleep 0.005; done; sleep 30; ip -6 r flush proto 57 Then look at the output of watch: at first, the number of routes grows regularly, and then, at some point, it will start jumping up and down. It should be possible to write a small program that automatically tests the presence of the bug (for instance by checking that the number of routes reported by "ip -6 r" is always increasing). So far, using the little shell script above, I saw the issue with: - Wheezy (3.2.0-4-amd64 + iproute 20120521-3+b3) The bug didn't show up with: - Wheezy with a multipath TCP kernel (3.14.27 + iproute 201407242100) - Archlinux (3.18.2 + iproute2 3.17.0 ) - Archlinux (3.14.27-1-mptcp + iproute2 3.18.0-1)
By the way, I think I found a kernel commit that should fix this issue: commit 1c2658545816088477e91860c3a645053719cb54 Author: Kumar Sundararajan <kumar@fb.com> Date: Thu Apr 24 09:48:53 2014 -0400 ipv6: fib: fix fib dump restart When the ipv6 fib changes during a table dump, the walk is restarted and the number of nodes dumped are skipped. But the existing code doesn't advance to the next node after a node is skipped. This can cause the dump to loop or produce lots of duplicates when the fib is modified during the dump. This change advances the walk to the next node if the current node is skipped after a restart. Signed-off-by: Kumar Sundararajan <kumar@fb.com> Signed-off-by: Chris Mason <clm@fb.com> Signed-off-by: David S. Miller <davem@davemloft.net> net/ipv6/ip6_fib.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) The patch is very simple... See also http://kernel.opensuse.org/cgit/kernel/commit/?id=1c2658545816088477e91860c3... Does Bird keep inserting routes in the kernel while it is scanning the routing table? If so, then we probably hit this bug. Thanks, Baptiste On Thu, Feb 05, 2015 at 02:50:08PM +0100, Baptiste Jonglez wrote:
On Sat, Jan 31, 2015 at 03:06:37PM +0100, Ondrej Zajicek wrote:
On Sat, Jan 31, 2015 at 02:47:51PM +0100, Baptiste Jonglez wrote:
This took several minutes to complete, and there certainly isn't so much IPv6 routes in the kernel: routes appear several times in the output of "ip -6 r". Running this command multiple times yields very different results each time.
Thus, I don't think the bug is in Bird. Could it be some kind of race condition with netlink? I haven't been able to find any reference to this bug, either in the kernel or in iproute2. For reference, this is on a Debian wheezy system, but I can reproduce the duplicate routes in "ip -6 r" on Debian jessie as well.
Interesting, what are the kernel versions in these Wheezy and Jessie systems? Does the problem (with 'ip -6 r') appears also when BIRD is not running?
So far, I saw this behaviour in Wheezy (3.2.0-4-amd64 + iproute 20120521-3+b3), and Jessie (3.14.4-1 + iproute2 3.16.0-2). In both cases, Bird was running.
I tried shutting down bird6 (with "persist" in the kernel protocol, so that routes stay in the kernel), and the problem seemed to persist:
$ ip -6 r | wc -l 26720 $ ip -6 r | wc -l 21794 $ ip -6 r | wc -l 50321 $ ip -6 r | wc -l 37602 $ ip -6 r | wc -l 59011
However, the amplitude is very reduced (when Bird is running and talking to the kernel, doing the same thing yields millions of routes).
I wonder what factors are specific to this problem. I remember there were a similar report or two few years ago, but these reports are too uncommon to be an universal problem in IPv6 Linux forwarding.
I actually found a way to reproduce this without Bird, just by inserting static routes. It seems that dumping the routing table while some routes are being inserted is enough to trigger the bug.
Simple way to see the issue: run this on one shell
watch -n 0.2 "ip -6 r | wc -l"
and run this in another, root, shell:
for i in {0000..3999}; do ip -6 r add unreachable 2001:db8:$i::/48 proto 57; sleep 0.005; done; sleep 30; ip -6 r flush proto 57
Then look at the output of watch: at first, the number of routes grows regularly, and then, at some point, it will start jumping up and down.
It should be possible to write a small program that automatically tests the presence of the bug (for instance by checking that the number of routes reported by "ip -6 r" is always increasing).
So far, using the little shell script above, I saw the issue with:
- Wheezy (3.2.0-4-amd64 + iproute 20120521-3+b3)
The bug didn't show up with:
- Wheezy with a multipath TCP kernel (3.14.27 + iproute 201407242100) - Archlinux (3.18.2 + iproute2 3.17.0 ) - Archlinux (3.14.27-1-mptcp + iproute2 3.18.0-1)
On Thu, Feb 05, 2015 at 03:07:07PM +0100, Baptiste Jonglez wrote:
By the way, I think I found a kernel commit that should fix this issue:
commit 1c2658545816088477e91860c3a645053719cb54 Author: Kumar Sundararajan <kumar@fb.com> Date: Thu Apr 24 09:48:53 2014 -0400
ipv6: fib: fix fib dump restart
When the ipv6 fib changes during a table dump, the walk is restarted and the number of nodes dumped are skipped. But the existing code doesn't advance to the next node after a node is skipped. This can cause the dump to loop or produce lots of duplicates when the fib is modified during the dump.
This change advances the walk to the next node if the current node is skipped after a restart.
Signed-off-by: Kumar Sundararajan <kumar@fb.com> Signed-off-by: Chris Mason <clm@fb.com> Signed-off-by: David S. Miller <davem@davemloft.net>
net/ipv6/ip6_fib.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
The patch is very simple... See also http://kernel.opensuse.org/cgit/kernel/commit/?id=1c2658545816088477e91860c3...
Does Bird keep inserting routes in the kernel while it is scanning the routing table? If so, then we probably hit this bug.
Hi Thanks for noticing it. We generally do not insert routes during the scan as scanning is an undivided process in BIRD and all synchronization is done immediately after the scan, but perhaps it is triggered anyways. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Mon, Feb 23, 2015 at 09:45:54PM +0100, Ondrej Zajicek wrote:
Does Bird keep inserting routes in the kernel while it is scanning the routing table? If so, then we probably hit this bug.
Hi
Thanks for noticing it.
We generally do not insert routes during the scan as scanning is an undivided process in BIRD and all synchronization is done immediately after the scan, but perhaps it is triggered anyways.
Ah, that's interesting. Since the box is a router, maybe new routes are inserted during the synchronisation because of the routing cache? That is, new destinations are looked up and inserted in the cache all the time by the kernel, so that the FIB is constantly modified. I'll try to disable the routing cache to see if it fixes the issue.
On Sat, Jan 31, 2015 at 12:32:44AM +0100, Baptiste Jonglez wrote:
I just tried downgrading from 1.4.5 to 1.4.4, using the 1.4.4-1~bpo70+1 Debian package from http://bird.network.cz/?download&tdir=debian/
The result is the same, bird6 also freezes periodically with version 1.4.4.
By the way, I think I ruled out the possibility that a particular BGP peer is sending garbage: the issue still arises when leaving only one BGP session active, whichever it is.
Is there anything else I can do to help troubleshoot the root cause of this issue?
Well, if you could run BIRD (presumably the current version) with 'ulimit -c unlimited' and then be able to kill BIRD during lockup using 'kill -QUIT <pid>', then resulting core dump would be highly useful for determining the source of the problem. But hitting the BIRD in the right moment could be problematic. In the next version, we will implement configurable 'watchdog' that would kill BIRD automatically during such lockup. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
participants (4)
-
Baptiste Jonglez -
Chris Caputo -
Ondrej Zajicek -
Pendzik, Edward