Package org.testng

Class TestNG

  • public class TestNG
extends java.lang.Object
    This class is the main entry point for running tests in the TestNG framework. Users can create their own TestNG object and invoke it in many different ways:
    • On an existing testng.xml
    • On a synthetic testng.xml, created entirely from Java
    • By directly setting the test classes
    You can also define which groups to include or exclude, assign parameters, etc...

    The command line parameters are:

    • -d outputdir: specify the output directory
    • -testclass class_name: specifies one or several class names
    • -testjar jar_name: specifies the jar containing the tests
    • -sourcedir src1;src2: ; separated list of source directories (used only when javadoc annotations are used)
    • -target
    • -groups
    • -testrunfactory
    • -listener

    Please consult documentation for more details.

    FIXME: should support more than simple paths for suite xmls

    See Also:
    usage()

    • Field Detail

      • LOGGER

        private static final Logger LOGGER
        This class' log4testng Logger.

      • DEFAULT_COMMAND_LINE_SUITE_NAME

        public static final java.lang.String DEFAULT_COMMAND_LINE_SUITE_NAME
        The default name for a suite launched from the command line
        See Also:
        Constant Field Values

      • DEFAULT_COMMAND_LINE_TEST_NAME

        public static final java.lang.String DEFAULT_COMMAND_LINE_TEST_NAME
        The default name for a test launched from the command line
        See Also:
        Constant Field Values

      • DEFAULT_THREADPOOL_FACTORY

        private static final java.lang.String DEFAULT_THREADPOOL_FACTORY
        See Also:
        Constant Field Values

      • DEFAULT_OUTPUTDIR

        public static final java.lang.String DEFAULT_OUTPUTDIR
        The default name of the result's output directory (keep public, used by Eclipse).
        See Also:
        Constant Field Values

      • m_instance

        private static TestNG m_instance

      • m_jCommander

        private static com.beust.jcommander.JCommander m_jCommander

      • m_commandLineMethods

        private java.util.List<java.lang.String> m_commandLineMethods

      • m_suites

        protected java.util.List<XmlSuite> m_suites

      • m_cmdlineSuites

        private java.util.List<XmlSuite> m_cmdlineSuites

      • m_outputDir

        private java.lang.String m_outputDir

      • m_includedGroups

        private java.lang.String[] m_includedGroups

      • m_excludedGroups

        private java.lang.String[] m_excludedGroups

      • m_isJUnit

        private java.lang.Boolean m_isJUnit

      • m_isMixed

        private java.lang.Boolean m_isMixed

      • m_useDefaultListeners

        protected boolean m_useDefaultListeners

      • m_failIfAllTestsSkipped

        private boolean m_failIfAllTestsSkipped

      • m_listenersToSkipFromBeingWiredIn

        private final java.util.List<java.lang.String> m_listenersToSkipFromBeingWiredIn

      • m_reporters

        private final java.util.Map<java.lang.Class<? extends IReporter>,​IReporter> m_reporters

      • DEFAULT_VERBOSE

        public static final java.lang.Integer DEFAULT_VERBOSE

      • m_threadCount

        private int m_threadCount

      • m_commandLineTestClasses

        private java.lang.Class<?>[] m_commandLineTestClasses

      • m_defaultSuiteName

        private java.lang.String m_defaultSuiteName

      • m_defaultTestName

        private java.lang.String m_defaultTestName

      • m_methodDescriptors

        private java.util.Map<java.lang.String,​java.lang.Integer> m_methodDescriptors

      • m_dataProviderThreadCount

        private java.lang.Integer m_dataProviderThreadCount

      • m_jarPath

        private java.lang.String m_jarPath

      • m_xmlPathInJar

        private java.lang.String m_xmlPathInJar
        The path of the testng.xml file inside the jar file

      • m_stringSuites

        private java.util.List<java.lang.String> m_stringSuites

      • m_end

        protected long m_end

      • m_start

        protected long m_start

      • m_isInitialized

        private boolean m_isInitialized

      • isSuiteInitialized

        private boolean isSuiteInitialized

      • m_verbose

        private java.lang.Integer m_verbose
        If m_verbose gets set, it will override the verbose setting in testng.xml

      • m_skipFailedInvocationCounts

        private java.lang.Boolean m_skipFailedInvocationCounts

      • m_testNames

        private java.util.List<java.lang.String> m_testNames
        The list of test names to run from the given suite

      • m_suiteThreadPoolSize

        private java.lang.Integer m_suiteThreadPoolSize

      • m_randomizeSuites

        private boolean m_randomizeSuites

      • m_alwaysRun

        private boolean m_alwaysRun

      • m_preserveOrder

        private java.lang.Boolean m_preserveOrder

      • m_groupByInstances

        private java.lang.Boolean m_groupByInstances

      • m_serviceLoaderClassLoader

        private java.net.URLClassLoader m_serviceLoaderClassLoader

    • Constructor Detail

      • TestNG

        public TestNG()
        Default constructor. Setting also usage of default listeners/reporters.

      • TestNG

        public TestNG​(boolean useDefaultListeners)
        Used by maven2 to have 0 output of any kind come out of testng.
        Parameters:
        useDefaultListeners - Whether or not any default reports should be added to tests.

    • Method Detail

      • init

        private void init​(boolean useDefaultListeners)

      • toggleFailureIfAllTestsWereSkipped

        public void toggleFailureIfAllTestsWereSkipped​(boolean failIfAllTestsSkipped)
        Parameters:
        failIfAllTestsSkipped - - Whether TestNG should enable/disable failing when all the tests were skipped and nothing was run (Mostly when a test is powered by a data provider and when the data provider itself fails causing all tests to skip).

      • setListenersToSkipFromBeingWiredInViaServiceLoaders

        public void setListenersToSkipFromBeingWiredInViaServiceLoaders​(java.lang.String... listeners)
        Parameters:
        listeners - - An array of fully qualified class names that should be skipped from being wired in via service loaders.

      • getStatus

        public int getStatus()

      • setOutputDirectory

        public void setOutputDirectory​(java.lang.String outputdir)
        Sets the output directory where the reports will be created.
        Parameters:
        outputdir - The directory.

      • setUseDefaultListeners

        public void setUseDefaultListeners​(boolean useDefaultListeners)
        Parameters:
        useDefaultListeners - If true before run(), the default listeners will not be used.
        • org.testng.reporters.TestHTMLReporter
        • org.testng.reporters.JUnitXMLReporter
        • org.testng.reporters.XMLReporter
        See Also:
        TestHTMLReporter, JUnitXMLReporter, XMLReporter

      • setTestJar

        public void setTestJar​(java.lang.String jarPath)
        Sets a jar containing a testng.xml file.
        Parameters:
        jarPath - - Path of the jar

      • setXmlPathInJar

        public void setXmlPathInJar​(java.lang.String xmlPathInJar)
        Parameters:
        xmlPathInJar - Sets the path to the XML file in the test jar file.

      • parseSuiteFiles

        private void parseSuiteFiles()

      • parseSuite

        private void parseSuite​(java.lang.String suitePath)

      • initializeSuitesAndJarFile

        public void initializeSuitesAndJarFile()

      • setThreadCount

        public void setThreadCount​(int threadCount)
        Parameters:
        threadCount - Define the number of threads in the thread pool.

      • setParallel

        @Deprecated
public void setParallel​(java.lang.String parallel)
        Deprecated.
        Use #setParallel(XmlSuite.ParallelMode) instead
        Parameters:
        parallel - Define whether this run will be run in parallel mode.

      • setCommandLineSuite

        public void setCommandLineSuite​(XmlSuite suite)

      • setTestClasses

        public void setTestClasses​(java.lang.Class[] classes)
        Set the test classes to be run by this TestNG object. This method will create a dummy suite that will wrap these classes called "Command Line Test".

        If used together with threadCount, parallel, groups, excludedGroups than this one must be set first.

        Parameters:
        classes - An array of classes that contain TestNG annotations.

      • splitMethod

        private java.lang.String[] splitMethod​(java.lang.String m)
        Given a string com.example.Foo.f1, return an array where [0] is the class and [1] is the method.

      • createCommandLineSuitesForMethods

        private java.util.List<XmlSuite> createCommandLineSuitesForMethods​(java.util.List<java.lang.String> commandLineMethods)
        Parameters:
        commandLineMethods - a string with the form "com.example.Foo.f1,com.example.Bar.f2"
        Returns:
        a list of XmlSuite objects that represent the list of classes and methods passed in parameter.

      • createCommandLineSuitesForClasses

        private java.util.List<XmlSuite> createCommandLineSuitesForClasses​(java.lang.Class[] classes)

      • addMethodSelector

        public void addMethodSelector​(java.lang.String className,
                              int priority)

      • setTestSuites

        public void setTestSuites​(java.util.List<java.lang.String> suites)
        Set the suites file names to be run by this TestNG object. This method tries to load and parse the specified TestNG suite xml files. If a file is missing, it is ignored.
        Parameters:
        suites - A list of paths to one more XML files defining the tests. For example: 
         TestNG tng = new TestNG();
 List<String> suites = Lists.newArrayList();
 suites.add("c:/tests/testng1.xml");
 suites.add("c:/tests/testng2.xml");
 tng.setTestSuites(suites);
 tng.run();

      • setXmlSuites

        public void setXmlSuites​(java.util.List<XmlSuite> suites)
        Specifies the XmlSuite objects to run.
        Parameters:
        suites - - The list of XmlSuite objects.
        See Also:
        XmlSuite

      • setExcludedGroups

        public void setExcludedGroups​(java.lang.String groups)
        Define which groups will be excluded from this run.
        Parameters:
        groups - A list of group names separated by a comma.

      • setGroups

        public void setGroups​(java.lang.String groups)
        Define which groups will be included from this run.
        Parameters:
        groups - A list of group names separated by a comma.

      • setTestRunnerFactoryClass

        private void setTestRunnerFactoryClass​(java.lang.Class testRunnerFactoryClass)

      • setObjectFactory

        public void setObjectFactory​(java.lang.Class c)

      • setListenerClasses

        public void setListenerClasses​(java.util.List<java.lang.Class<? extends ITestNGListener>> classes)
        Define which listeners to user for this run.
        Parameters:
        classes - A list of classes, which must be either ISuiteListener, ITestListener or IReporter

      • addListener

        @Deprecated
public void addListener​(java.lang.Object listener)
        Deprecated.
        Use addListener(ITestNGListener) instead
        Parameters:
        listener - The listener to add

      • maybeAddListener

        private static <E> void maybeAddListener​(java.util.Map<java.lang.Class<? extends E>,​E> map,
                                         E value)

      • maybeAddListener

        private static <E> void maybeAddListener​(java.util.Map<java.lang.Class<? extends E>,​E> map,
                                         java.lang.Class<? extends E> type,
                                         E value,
                                         boolean quiet)

      • getReporters

        public java.util.Set<IReporter> getReporters()

      • getTestListeners

        public java.util.List<ITestListener> getTestListeners()

      • getSuiteListeners

        public java.util.List<ISuiteListener> getSuiteListeners()

      • setVerbose

        public void setVerbose​(int verbose)
        Sets the level of verbosity. This value will override the value specified in the test suites.
        Parameters:
        verbose - the verbosity level (0 to 10 where 10 is most detailed) Actually, this is a lie: you can specify -1 and this will put TestNG in debug mode (no longer slicing off stack traces and all).

      • setExecutorFactoryClass

        public void setExecutorFactoryClass​(java.lang.String clazzName)

      • createExecutorFactoryInstanceUsing

        private IExecutorFactory createExecutorFactoryInstanceUsing​(java.lang.String clazzName)

      • setExecutorFactory

        public void setExecutorFactory​(IExecutorFactory factory)

      • initializeCommandLineSuites

        private void initializeCommandLineSuites()

      • initializeCommandLineSuitesParams

        private void initializeCommandLineSuitesParams()

      • initializeCommandLineSuitesGroups

        private void initializeCommandLineSuitesGroups()

      • initializeCommandLineSuitesGroups

        private static void initializeCommandLineSuitesGroups​(XmlSuite s,
                                                      boolean hasIncludedGroups,
                                                      java.lang.String[] m_includedGroups,
                                                      boolean hasExcludedGroups,
                                                      java.lang.String[] m_excludedGroups)

      • addReporter

        private void addReporter​(java.lang.Class<? extends IReporter> r)

      • initializeDefaultListeners

        private void initializeDefaultListeners()

      • initializeConfiguration

        private void initializeConfiguration()

      • addListeners

        private void addListeners​(XmlSuite s)

      • addServiceLoaderListeners

        private void addServiceLoaderListeners()
        Using reflection to remain Java 5 compliant.

      • sanityCheck

        private void sanityCheck()
        Before suites are executed, do a sanity check to ensure all required conditions are met. If not, throw an exception to stop test execution
        Throws:
        TestNGException - if the sanity check fails

      • initializeEverything

        public void initializeEverything()
        Invoked by the remote runner.

      • run

        public void run()
        Run TestNG.

      • runSuites

        protected java.util.List<ISuite> runSuites()
        Run the test suites.

        This method can be overridden by subclass.
        For example, DistributedTestNG to run in master/slave mode according to commandline args.

        Returns:
        - List of suites that were run as ISuite objects.
        Since:
        6.9.11 when moving distributed/remote classes out into separate project

      • runSuiteAlterationListeners

        private void runSuiteAlterationListeners()

      • runExecutionListeners

        private void runExecutionListeners​(boolean start)

      • usage

        private static void usage()

      • generateReports

        private void generateReports​(java.util.List<ISuite> suiteRunners)

      • runSuitesLocally

        public java.util.List<ISuite> runSuitesLocally()
        This needs to be public for maven2, for now..At least until an alternative mechanism is found.
        Returns:
        The locally run suites

      • error

        private static void error​(java.lang.String s)

      • getVerbose

        private int getVerbose​(XmlSuite xmlSuite)
        Returns:
        the verbose level, checking in order: the verbose level on the suite, the verbose level on the TestNG object, or 1.

      • runSuitesSequentially

        private void runSuitesSequentially​(XmlSuite xmlSuite,
                                   SuiteRunnerMap suiteRunnerMap,
                                   int verbose,
                                   java.lang.String defaultSuiteName)
        Recursively runs suites. Runs the children suites before running the parent suite. This is done so that the results for parent suite can reflect the combined results of the children suites.
        Parameters:
        xmlSuite - XML Suite to be executed
        suiteRunnerMap - Maps XmlSuites to respective ISuite
        verbose - verbose level
        defaultSuiteName - default suite name

      • populateSuiteGraph

        private void populateSuiteGraph​(IDynamicGraph<ISuite> suiteGraph,
                                SuiteRunnerMap suiteRunnerMap,
                                XmlSuite xmlSuite)
        Populates the dynamic graph with the reverse hierarchy of suites. Edges are added pointing from child suite runners to parent suite runners, hence making parent suite runners dependent on all the child suite runners
        Parameters:
        suiteGraph - dynamic graph representing the reverse hierarchy of SuiteRunners
        suiteRunnerMap - Map with XMLSuite as key and its respective SuiteRunner as value
        xmlSuite - XML Suite

      • createSuiteRunners

        private void createSuiteRunners​(SuiteRunnerMap suiteRunnerMap,
                                XmlSuite xmlSuite)
        Creates the SuiteRunners and populates the suite runner map with this information
        Parameters:
        suiteRunnerMap - Map with XMLSuite as key and it's respective SuiteRunner as value. This is updated as part of this method call
        xmlSuite - Xml Suite (and its children) for which SuiteRunners are created

      • createSuiteRunner

        private SuiteRunner createSuiteRunner​(XmlSuite xmlSuite)
        Creates a suite runner and configures its initial state

      • main

        public static void main​(java.lang.String[] argv)
        The TestNG entry point for command line execution.
        Parameters:
        argv - the TestNG command line parameters.

      • privateMain

        public static TestNG privateMain​(java.lang.String[] argv,
                                 ITestListener listener)
        Note: this method is not part of the public API and is meant for internal usage only.
        Parameters:
        argv - The param arguments
        listener - The listener
        Returns:
        The TestNG instance

      • configure

        protected void configure​(CommandLineArgs cla)
        Configure the TestNG instance based on the command line parameters.
        Parameters:
        cla - The command line parameters

      • setSuiteThreadPoolSize

        public void setSuiteThreadPoolSize​(java.lang.Integer suiteThreadPoolSize)

      • getSuiteThreadPoolSize

        public java.lang.Integer getSuiteThreadPoolSize()

      • setRandomizeSuites

        public void setRandomizeSuites​(boolean randomizeSuites)

      • alwaysRunListeners

        public void alwaysRunListeners​(boolean alwaysRun)

      • setSourcePath

        @Deprecated
public void setSourcePath​(java.lang.String path)
        Deprecated.
        This method is invoked by Maven's Surefire, only remove it once Surefire has been modified to no longer call it.
        Parameters:
        path - The path

      • parseInt

        private static int parseInt​(java.lang.Object value)

      • configure

        @Deprecated
public void configure​(java.util.Map cmdLineArgs)
        Deprecated.
        use new configure(CommandLineArgs) method
        This method is invoked by Maven's Surefire to configure the runner, do not remove unless you know for sure that Surefire has been updated to use the new configure(CommandLineArgs) method.
        Parameters:
        cmdLineArgs - The command line

      • setTestNames

        public void setTestNames​(java.util.List<java.lang.String> testNames)
        Parameters:
        testNames - Only run the specified tests from the suite.

      • setSkipFailedInvocationCounts

        public void setSkipFailedInvocationCounts​(java.lang.Boolean skip)

      • addReporter

        private void addReporter​(ReporterConfig reporterConfig)

      • setJUnit

        public void setJUnit​(java.lang.Boolean isJUnit)
        Specify if this run should be made in JUnit mode
        Parameters:
        isJUnit - - Specify if this run should be made in JUnit mode

      • setMixed

        public void setMixed​(java.lang.Boolean isMixed)
        Parameters:
        isMixed - Specify if this run should be made in mixed mode

      • validateCommandLineParameters

        protected static void validateCommandLineParameters​(CommandLineArgs args)
        Double check that the command line parameters are valid.
        Parameters:
        args - The command line to check

      • hasFailure

        public boolean hasFailure()
        Returns:
        true if at least one test failed.

      • hasFailureWithinSuccessPercentage

        public boolean hasFailureWithinSuccessPercentage()
        Returns:
        true if at least one test failed within success percentage.

      • hasSkip

        public boolean hasSkip()
        Returns:
        true if at least one test was skipped.

      • exitWithError

        static void exitWithError​(java.lang.String msg)

      • getOutputDirectory

        public java.lang.String getOutputDirectory()

      • getDefaultSuiteName

        public java.lang.String getDefaultSuiteName()
        Returns:
        the defaultSuiteName

      • setDefaultSuiteName

        public void setDefaultSuiteName​(java.lang.String defaultSuiteName)
        Parameters:
        defaultSuiteName - the defaultSuiteName to set

      • getDefaultTestName

        public java.lang.String getDefaultTestName()
        Returns:
        the defaultTestName

      • setDefaultTestName

        public void setDefaultTestName​(java.lang.String defaultTestName)
        Parameters:
        defaultTestName - the defaultTestName to set

      • setConfigFailurePolicy

        public void setConfigFailurePolicy​(XmlSuite.FailurePolicy failurePolicy)
        Sets the policy for whether or not to ever invoke a configuration method again after it has failed once. Possible values are defined in XmlSuite. The default value is XmlSuite.FailurePolicy.SKIP
        Parameters:
        failurePolicy - the configuration failure policy

      • getConfigFailurePolicy

        public XmlSuite.FailurePolicy getConfigFailurePolicy()
        Returns the configuration failure policy.
        Returns:
        config failure policy

      • getDefault

        @Deprecated
public static TestNG getDefault()
        Deprecated.
        since 5.1
        Returns:
        The default instance

      • setConfigurable

        private void setConfigurable​(IConfigurable c)

      • setHookable

        private void setHookable​(IHookable h)

      • setMethodInterceptor

        public void setMethodInterceptor​(IMethodInterceptor methodInterceptor)

      • setDataProviderThreadCount

        public void setDataProviderThreadCount​(int count)

      • addClassLoader

        public void addClassLoader​(java.lang.ClassLoader loader)
        Add a class loader to the searchable loaders.
        Parameters:
        loader - The class loader to add

      • setPreserveOrder

        public void setPreserveOrder​(boolean b)

      • getStart

        protected long getStart()

      • getEnd

        protected long getEnd()

      • setGroupByInstances

        public void setGroupByInstances​(boolean b)

      • setServiceLoaderClassLoader

        public void setServiceLoaderClassLoader​(java.net.URLClassLoader ucl)

      • addServiceLoaderListener

        private void addServiceLoaderListener​(ITestNGListener l)

      • getServiceLoaderListeners

        public java.util.List<ITestNGListener> getServiceLoaderListeners()

      • setInjectorFactory

        public void setInjectorFactory​(IInjectorFactory factory)