Hello all, is there something like a full (regression) test suite for BIRD that one could run after building BIRD 2.0.x? Regards, Robert
Hello! We don't know about any, sorry. It is something we'd like to create and maintain. If you are into helping us with this issue, you're very welcome. Maria On April 26, 2019 12:35:00 AM GMT+02:00, Robert Scheck <bird@robert-scheck.de> wrote:
Hello all,
is there something like a full (regression) test suite for BIRD that one could run after building BIRD 2.0.x?
Regards, Robert
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Hello Maria, On Fri, 26 Apr 2019, Maria Matějka wrote:
We don't know about any, sorry. It is something we'd like to create and maintain. If you are into helping us with this issue, you're very welcome.
is there currently any test suite or are there any tests that you e.g. run before doing a new BIRD release? Regards, Robert
We have a test environment (much limited) for running basic configuration patterns and a suite of basic unit tests. This is, anyway, not really enough. We specifically do not check protocols much, especially corner cases are not covered at all. And we know that it is bad, we are also hiring a tester now (hoping that money will be sufficient to keep them) to have a dedicated person just for testing. Currently, all the testing is squeezed between another development quests. I also think that there is still much work to help with even when we have a tester. Shall I try to write some more detailed summary of how the testing works now and what you could help with? Is it something you would be interested in? Thanks Maria PS. Other way to help us with testing is to pay for a support contract or sponsor the test development directly. On April 26, 2019 1:05:18 AM GMT+02:00, Robert Scheck <bird@robert-scheck.de> wrote:
Hello Maria,
On Fri, 26 Apr 2019, Maria Matějka wrote:
We don't know about any, sorry. It is something we'd like to create and maintain. If you are into helping us with this issue, you're very welcome.
is there currently any test suite or are there any tests that you e.g. run before doing a new BIRD release?
Regards, Robert
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Hello Maria, On Fri, 26 Apr 2019, Maria Matějka wrote:
We have a test environment (much limited) for running basic configuration patterns and a suite of basic unit tests.
is somewhere documented how to run this one? For the very first step, I would like to run it as part of the RPM builds at Fedora and EPEL on all build architectures (currently i686, x86_64, armv7hl, aarch64, ppc64le and s390x) to see where we end up at the moment. Regards, Robert
On Fri, Apr 26, 2019 at 11:16:09AM +0200, Robert Scheck wrote:
Hello Maria,
On Fri, 26 Apr 2019, Maria Matějka wrote:
We have a test environment (much limited) for running basic configuration patterns and a suite of basic unit tests.
is somewhere documented how to run this one? For the very first step, I would like to run it as part of the RPM builds at Fedora and EPEL on all build architectures (currently i686, x86_64, armv7hl, aarch64, ppc64le and s390x) to see where we end up at the moment.
Hi The suite of basic unit tests can be run by 'make test'. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hello Ondrej, hello Maria, On Fri, 26 Apr 2019, Ondrej Zajicek wrote:
The suite of basic unit tests can be run by 'make test'.
that one fails for BIRD 2.0.4 on RHEL/CentOS 7 on ppc64 (big endian) like this: [...] obj/lib/fletcher16_test: t_fletcher16_compute [FAIL] obj/lib/fletcher16_test: t_fletcher16_checksum [FAIL] obj/lib/fletcher16_test [FAIL] https://kojipkgs.fedoraproject.org//work/tasks/1532/34501532/build.log for build logs. https://koji.fedoraproject.org/koji/taskinfo?taskID=34501529 in case you would like to compare output of different architectures. And: It also fails on Fedora 31 on s390x in the same way like for ppc64. https://kojipkgs.fedoraproject.org//work/tasks/1613/34501613/build.log and https://koji.fedoraproject.org/koji/taskinfo?taskID=34501608 for details. Just let me know if I can be helpful somehow in tracking them down, even my gdb knowledge is quite limited. Regards, Robert
Hello!
On Fri, 26 Apr 2019, Ondrej Zajicek wrote:
The suite of basic unit tests can be run by 'make test'.
that one fails for BIRD 2.0.4 on RHEL/CentOS 7 on ppc64 (big endian) like this:
[...] obj/lib/fletcher16_test: t_fletcher16_compute [FAIL] obj/lib/fletcher16_test: t_fletcher16_checksum [FAIL] obj/lib/fletcher16_test [FAIL]
Could you please: 1) run $ make clean && make obj/lib/fletcher16_test and send us all the warnings you find there 2) run $ obj/lib/fletcher16_test -vvv and send us the output to narrow down what is happening there
https://kojipkgs.fedoraproject.org//work/tasks/1532/34501532/build.log for build logs. https://koji.fedoraproject.org/koji/taskinfo?taskID=34501529 in case you would like to compare output of different architectures.
And: It also fails on Fedora 31 on s390x in the same way like for ppc64.
https://kojipkgs.fedoraproject.org//work/tasks/1613/34501613/build.log and https://koji.fedoraproject.org/koji/taskinfo?taskID=34501608 for details.
Bah! What a s***load of warnings! I hope that I manage going through them on Monday. Thank you for reporting this. Maria
Hello Maria, On Sat, 27 Apr 2019, Maria Matejka wrote:
Could you please: 1) run $ make clean && make obj/lib/fletcher16_test and send us all the warnings you find there
2) run $ obj/lib/fletcher16_test -vvv and send us the output to narrow down what is happening there
there we go (not sure what you would like to achieve with "make clean", each Fedora/EPEL build runs in a fresh chroot environment anyway), but you can find the desired commands in the build.log (search for the first "%check") below. Just grab the build logs within a couple of days, the garbage collector will clean up them sooner or later. ppc64: https://kojipkgs.fedoraproject.org//work/tasks/2839/34502839/build.log https://koji.fedoraproject.org/koji/taskinfo?taskID=34502836 s390x: https://kojipkgs.fedoraproject.org//work/tasks/2874/34502874/build.log https://koji.fedoraproject.org/koji/taskinfo?taskID=34502869
Thank you for reporting this.
You're welcome. Let me know if you need something else or other builds or parameters etc. Regards, Robert
On Sat, Apr 27, 2019 at 10:53:44PM +0200, Robert Scheck wrote:
Hello Ondrej, hello Maria,
On Fri, 26 Apr 2019, Ondrej Zajicek wrote:
The suite of basic unit tests can be run by 'make test'.
that one fails for BIRD 2.0.4 on RHEL/CentOS 7 on ppc64 (big endian) like this:
[...] obj/lib/fletcher16_test: t_fletcher16_compute [FAIL] obj/lib/fletcher16_test: t_fletcher16_checksum [FAIL] obj/lib/fletcher16_test [FAIL]
https://kojipkgs.fedoraproject.org//work/tasks/1532/34501532/build.log for build logs. https://koji.fedoraproject.org/koji/taskinfo?taskID=34501529 in case you would like to compare output of different architectures.
And: It also fails on Fedora 31 on s390x in the same way like for ppc64.
https://kojipkgs.fedoraproject.org//work/tasks/1613/34501613/build.log and https://koji.fedoraproject.org/koji/taskinfo?taskID=34501608 for details.
Just let me know if I can be helpful somehow in tracking them down, even my gdb knowledge is quite limited.
Hi Seems like you compile it with your CFLAGS. There are compiler options which we expect there to be: -fno-strict-aliasing -fno-strict-overflow Although it probably is not relevant to this test, BIRD should always be build with these. Also, could you try run the test manually: ./obj/lib/fletcher16_test -vv ? It is possible that it is just a bug in the test, we should add there some explicit test vectors instead of comparison with straightforward implementation. Also, if you could add these options (which we usually use): -Wno-parentheses -Wno-pointer-sign -Wno-missing-field-initializers It would eliminate plenty of irrelevant build warnings. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hello Ondrej, On Tue, 30 Apr 2019, Ondrej Zajicek wrote:
Seems like you compile it with your CFLAGS. There are compiler options which we expect there to be: -fno-strict-aliasing -fno-strict-overflow
well, Fedora just exports its own CFLAGS before ./configure. So, if BIRD depends on some compiler options, ./configure should explicitly check for the existence and append them as necessary (like other projects do).
Also, could you try run the test manually:
./obj/lib/fletcher16_test -vv
I am sorry, I can not run interactive commands on the Fedora builders, just the commands from the build instructions that come along with a test build; thus arbitrary unprivileged commands in general, but no interactive ones. https://kojipkgs.fedoraproject.org/work/tasks/1279/34611279/build.log with your requested CFLAGS, run as part of "make test", run separately and run using strace. https://koji.fedoraproject.org/koji/taskinfo?taskID=34611276 if you would like to compare the build logs with other architectures. Alternatively, I could ask for access to s390x and ppc64 systems at Fedora, but this will be a manual process and we likely have to follow-up this off- list then. Just let me know in case you think it could be helpful - and you would like to walk that path (this would lead you to an interactive shell). Regards, Robert
participants (4)
-
Maria Matejka -
Maria Matějka
-
Ondrej Zajicek -
Robert Scheck