[testng-users] TestNG throws a misleading error when @Factory method returns empty array.

[Jorn Vernee]

Hi,

I was testing using a testng test akin to the following:

import org.testng.TestNG;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class Main {

public static class MyTest {

private final int i;

public MyTest(int i) {
this.i = i;
}

@Test
public void testDownCall() {
System.out.println("test: " + i);
}

}

@Factory
public Object[] getTests() throws Throwable {
return generateTests();
}

private Object[] generateTests() {
return new Object[] {}; // erroneously returning empty array
}

public static void main(String[] args) throws Throwable {
TestNG t = new TestNG();
t.setTestClasses(new Class<?>[] { Main.class });
t.run();
}

}

Running this with the 7.0.0-beta1 version of TestNG I get the following
error:

Exception in thread "main" org.testng.TestNGException:
An error occurred while instantiating class test.Main$MyTest. Check to make
sure it can be instantiated
at org.testng.internal.ClassHelper.createInstance1(ClassHelper.java:388)
at org.testng.internal.ClassHelper.createInstance(ClassHelper.java:336)
at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:110)
at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:188)
at org.testng.TestClass.getInstances(TestClass.java:94)
at org.testng.TestClass.initTestClassesAndInstances(TestClass.java:79)
at org.testng.TestClass.init(TestClass.java:71)
at org.testng.TestClass.<init>(TestClass.java:36)
at org.testng.TestRunner.initMethods(TestRunner.java:461)
at org.testng.TestRunner.init(TestRunner.java:340)
at org.testng.TestRunner.init(TestRunner.java:293)
at org.testng.TestRunner.<init>(TestRunner.java:178)
at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(
SuiteRunner.java:603)
at org.testng.SuiteRunner.init(SuiteRunner.java:194)
at org.testng.SuiteRunner.<init>(SuiteRunner.java:125)
at org.testng.TestNG.createSuiteRunner(TestNG.java:1230)
at org.testng.TestNG.createSuiteRunners(TestNG.java:1209)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1058)
at org.testng.TestNG.runSuites(TestNG.java:997)
at org.testng.TestNG.run(TestNG.java:965)
at test.Main.main(Main.java:32)

I spent a few days puzzled with this, since the test was working fine
before. But it turned out that the method generating the test classes
(which in reality is quite a bit more complex) was suddenly returning an
empty array.

Maybe a warning or error should be emitted when an @Factory method returns
an empty array? Instead of silently falling back on trying to construct
test instances automatically.

[Krishnan Mahadevan]

Jorn,



I recently fixed an issue around this as part of addressing : https://github.com/cbeust/testng/issues/1924



The fix is available in the latest snapshot version of TestNG.



Can you please quickly try using TestNG 7.0.0-SNAPSHOT and let us know how it goes?

The sample you shared here, works fine when I run it against the master branch of the TestNG codebase.



The URL that you may need to add in your pom file under <repositories> section is: https://oss.sonatype.org/content/repositories/snapshots



Thanks & Regards

Krishnan Mahadevan



"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/



From: <testng-***@googlegroups.com> on behalf of Jorn Vernee <***@xs4all.nl>
Reply-To: <testng-***@googlegroups.com>
Date: Wednesday, November 14, 2018 at 3:05 AM
To: testng-users <testng-***@googlegroups.com>
Subject: [testng-users] TestNG throws a misleading error when @Factory method returns empty array.



Hi,



I was testing using a testng test akin to the following:



import org.testng.TestNG;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class Main {

public static class MyTest {

private final int i;

public MyTest(int i) {
this.i = i;
}

@Test
public void testDownCall() {
System.out.println("test: " + i);
}

}

@Factory
public Object[] getTests() throws Throwable {
return generateTests();
}

private Object[] generateTests() {
return new Object[] {}; // erroneously returning empty array
}

public static void main(String[] args) throws Throwable {
TestNG t = new TestNG();
t.setTestClasses(new Class<?>[] { Main.class });
t.run();
}

}


Running this with the 7.0.0-beta1 version of TestNG I get the following error:



Exception in thread "main" org.testng.TestNGException:
An error occurred while instantiating class test.Main$MyTest. Check to make sure it can be instantiated
at org.testng.internal.ClassHelper.createInstance1(ClassHelper.java:388)
at org.testng.internal.ClassHelper.createInstance(ClassHelper.java:336)
at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:110)
at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:188)
at org.testng.TestClass.getInstances(TestClass.java:94)
at org.testng.TestClass.initTestClassesAndInstances(TestClass.java:79)
at org.testng.TestClass.init(TestClass.java:71)
at org.testng.TestClass.<init>(TestClass.java:36)
at org.testng.TestRunner.initMethods(TestRunner.java:461)
at org.testng.TestRunner.init(TestRunner.java:340)
at org.testng.TestRunner.init(TestRunner.java:293)
at org.testng.TestRunner.<init>(TestRunner.java:178)
at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:603)
at org.testng.SuiteRunner.init(SuiteRunner.java:194)
at org.testng.SuiteRunner.<init>(SuiteRunner.java:125)
at org.testng.TestNG.createSuiteRunner(TestNG.java:1230)
at org.testng.TestNG.createSuiteRunners(TestNG.java:1209)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1058)
at org.testng.TestNG.runSuites(TestNG.java:997)
at org.testng.TestNG.run(TestNG.java:965)
at test.Main.main(Main.java:32)



I spent a few days puzzled with this, since the test was working fine before. But it turned out that the method generating the test classes (which in reality is quite a bit more complex) was suddenly returning an empty array.



Maybe a warning or error should be emitted when an @Factory method returns an empty array? Instead of silently falling back on trying to construct test instances automatically.
--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users+***@googlegroups.com.
To post to this group, send email to testng-***@googlegroups.com.
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Jorn Vernee]

Hi Krishnan,

I've tried the 7.0.0-SNAPSHOT from that sonartype repo but that gives the
same error.

I'm trying to build the testng repo, but I get a
`java.lang.NoClassDefFoundError: com/beust/jcommander/ParameterException`
with the resulting testng-7.0.0-SNAPSHOT.jar from `gradle jar`. (I don't
use gradle much, so I might be missing something)

I've looked at the issue you've linked, but that person's @Factory method
is not returning an empty array, so I don't think it's the same use-case?

I'm thinking the code that invokes the factory method [1] could do a check
to see if the returned array is empty, and emit a warning on stderr or
something like that.

[1] :
https://github.com/cbeust/testng/blob/bbeafd15cc234e8ba983311d97b11f6becabee0d/src/main/java/org/testng/internal/FactoryMethod.java#L168

[⇜Krishnan Mahadevan⇝]

Jorne,

Ok I got the issue. I was thinking of something else as the problem. Didn't
realize that you were basically having an issue with the error message that
TestNG was throwing. My apologies for the confusion.

Would you like to raise a PR for this (Since you have spent time trying to
figure out where the issue is stemming from) ?
If yes, please file an issue and help create a PR.

If not, please let me know. I will get to this at the earliest.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening
or in love with someone else!"

Post by Jorn Vernee
Hi Krishnan,
I've tried the 7.0.0-SNAPSHOT from that sonartype repo but that gives the
same error.
I'm trying to build the testng repo, but I get a
`java.lang.NoClassDefFoundError: com/beust/jcommander/ParameterException`
with the resulting testng-7.0.0-SNAPSHOT.jar from `gradle jar`. (I don't
use gradle much, so I might be missing something)
is not returning an empty array, so I don't think it's the same use-case?
I'm thinking the code that invokes the factory method [1] could do a check
to see if the returned array is empty, and emit a warning on stderr or
something like that.
https://github.com/cbeust/testng/blob/bbeafd15cc234e8ba983311d97b11f6becabee0d/src/main/java/org/testng/internal/FactoryMethod.java#L168
--
You received this message because you are subscribed to the Google Groups
"testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Jorn Vernee]

I'm afraid I don't really have time to make a PR right now.

I'm also not 100% sure what the correct response from testng should be. I'm
in favor of throwing an exception at that point, since a vacuously passing
test is an error to me, but I'm not sure how you feel about this use case?

To give a better idea of my use case; we are generating native sources and
then crawling those to generate a java artifact, then I'm iterating the
methods in that artifact to create tests out of in the @Factory method. The
problem in this case was with the script that generates the native sources,
which was erroneously generating empty files, so the java artifact ended up
empty and so did the array returned by the @Factory method.

I should have checked the other links in the chain as well, but the error I
got from testng really threw me off. I thought we had a problem with the
build system not updating the test files, or maybe some issue with the
tests referencing classes loaded by different class loaders, or maybe some
issue with the testing framework used (jtreg) launching tests in separate
VMs.

[⇜Krishnan Mahadevan⇝]

Throwing an exception and aborting the execution sounds like the right
think to.

We do have a similar edit check in terms of a data provider. Guess we can
add it here as well.

I will take care of having this fixed at the earliest.

Post by Jorn Vernee
I'm afraid I don't really have time to make a PR right now.
I'm also not 100% sure what the correct response from testng should be.
I'm in favor of throwing an exception at that point, since a vacuously
passing test is an error to me, but I'm not sure how you feel about this
use case?
To give a better idea of my use case; we are generating native sources and
then crawling those to generate a java artifact, then I'm iterating the
problem in this case was with the script that generates the native sources,
which was erroneously generating empty files, so the java artifact ended up
I should have checked the other links in the chain as well, but the error
I got from testng really threw me off. I thought we had a problem with the
build system not updating the test files, or maybe some issue with the
tests referencing classes loaded by different class loaders, or maybe some
issue with the testing framework used (jtreg) launching tests in separate
VMs.
--
You received this message because you are subscribed to the Google Groups
"testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening
or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

[Krishnan Mahadevan]

Jorn,



This issue is being tracked as part of https://github.com/cbeust/testng/issues/1953.

I have fixed this issue and the changes should be available for you as part of TestNG 7.0.0-SNAPSHOT if you want to give it a try immediately, and as part of TestNG 7.0.0 once it has been released.





Thanks & Regards

Krishnan Mahadevan



"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/



From: ⇜Krishnan Mahadevan⇝ <***@gmail.com>
Date: Wednesday, November 14, 2018 at 4:39 PM
To: <testng-***@googlegroups.com>
Subject: Re: [testng-users] TestNG throws a misleading error when @Factory method returns empty array.



Throwing an exception and aborting the execution sounds like the right think to.



We do have a similar edit check in terms of a data provider. Guess we can add it here as well.



I will take care of having this fixed at the earliest.





On Wed 14 Nov, 2018, 16:35 Jorn Vernee, <***@xs4all.nl> wrote:

I'm afraid I don't really have time to make a PR right now.



I'm also not 100% sure what the correct response from testng should be. I'm in favor of throwing an exception at that point, since a vacuously passing test is an error to me, but I'm not sure how you feel about this use case?



To give a better idea of my use case; we are generating native sources and then crawling those to generate a java artifact, then I'm iterating the methods in that artifact to create tests out of in the @Factory method. The problem in this case was with the script that generates the native sources, which was erroneously generating empty files, so the java artifact ended up empty and so did the array returned by the @Factory method.



I should have checked the other links in the chain as well, but the error I got from testng really threw me off. I thought we had a problem with the build system not updating the test files, or maybe some issue with the tests referencing classes loaded by different class loaders, or maybe some issue with the testing framework used (jtreg) launching tests in separate VMs.
--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users+***@googlegroups.com.
To post to this group, send email to testng-***@googlegroups.com.
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
--
Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/