Test suite for MagicMirror²

roramirez

Hi, I recently pushed a Pull Request with two e2e testing.
https://github.com/MichMich/MagicMirror/pull/669

These use their own configurations like @qistoph mention and after added the MM_CONFIG_FILE variable for MagicMirror.

It’s really easy create new test for check a different part of MagicMirror and their modules.

strawberry 3.141

@roramirez what is the reason for a 10 sec timeout in each test? If there are a many tests this will sum up

roramirez

@strawberry-3.141 said in Test suite for MagicMirror²:

what is the reason for a 10 sec timeout in each test? If there are a many tests this will sum up

It’s does not accumulate. It’s a timeout, the maximun time for the test. You can set different timeout for every test.

morozgrafix

I haven’t looked thoroughly at the code and I wanted to suggest setting a default timeout that can be overwritten in individual tests as needed. It may make test code a little bit cleaner and less repetitive.

Good job getting it going!

roramirez

@morozgrafix Yes, I knowed is not more cleaner and repetitive but was the first proof of concept ;)

Good idea set default timeout. I added two new task in Trello board about you mentioned.

morozgrafix

@roramirez I’ve played a little with test suite and added basic test for clock module. Then I realized that if we want to test different config options for a given module we may need to have to create multiple configs, which may be challenging with current tests directory organization.
Also I believe that process.env.MM_CONFIG_FILE = "tests/confs/config_name.js" line needs to be moved into before step (seems to work there or it can go into beforeEach if needed). Otherwise when running npm run test:e2e first config that gets picked up seems to persist throughout all of the tests and always used by the app.js.

I was wondering if structure similar to this makes sense:

tests
├── configs
│   ├── env.js
│   └── modules
│   │   └── clock
│   │   │   └── clock_12hr.js
│   │   │   └── clock_24hr.js
│   │   │   └── clock_analog.js
│   │   └── helloworld
│   │       └── helloworld.js
├── e2e
│   ├── env_spec.js
│   └── modules
│       ├── clock_spec.js
│       └── helloworld_spec.js
├── functions
│   └── compare-version_spec.js
└── global_vars
    └── root_path.js

but this would involve changing app.use("/tests/confs", express.static(path.resolve(global.root_path + "/tests/confs"))); in the server.js to be somewhat dynamic and I’m not very familiar with express on how it can be done. I think that’s easily solved by just changing it to app.use("/tests/configs", express.static(path.resolve(global.root_path + "/tests/configs")));

roramirez

@morozgrafix Seem good sense going to a refactor of the structure.

I think we can take two way acord you mentioned.

1.- Include your test for clock module with new format for name and respective directories
2.- Do it the refactor to all tests that remain.

I really like see how you resolve the multiples instances configuration in a one tests file.

morozgrafix

@roramirez cool thanks. I have it all ready on my fork. Will submit PR for review.

qistoph

Two commits I’ve worked on for the testing.

Check keys in the translation files. Produces errors currently, so I’ve added .skip.
https://github.com/qistoph/MagicMirror/commit/123392c54934e49a397d586c1fb8dbcc4cc5d12b
To prevent loading app.js from corrupting the Mocha test environment, I suggest to execute the app.js in a virtual environment. This can also serve as an example for future test cases where code needs to be executed in the global namespace.
https://github.com/qistoph/MagicMirror/commit/cd8bee1371ffc6cce7b7bf44f85cd03705e4c1bd

Any thoughts before I submit a PR?

strawberry 3.141

@qistoph i like the idea of the sandbox

is there a way to just warn instead of error for missing translation keys?

qistoph

@strawberry-3.141
How about this?
https://github.com/qistoph/MagicMirror/commit/406ae4e8c37cbf7e31c89f5341d7715bacbcf0d2

			try {
				expect(fileKeys).to.deep.equal(baseKeys);
			} catch(e) {
				if (e instanceof chai.AssertionError) {
					this.skip();
				} else {
					throw e;
				}
			}

roramirez

@qistoph Nice work. Related with your first point there a discussion about translations of JSON file.
https://github.com/MichMich/MagicMirror/pull/679

roramirez

Hi Guys,

Today I’ve playing with test suite I something going me crazy

I create the follow test
https://github.com/roramirez/MagicMirror/commit/88f1f6bbd4fe6d7f3b9391993d207c7b9ebfb2e9

Two check on same test case. On my enviroment ever but ever give me fail

smirror :: ~/MagicMirror ‹without-modules-test*› » ./node_modules/mocha/bin/mocha tests/e2e/without_modules.js


  Check configuration without modules
    ✓ Show the message create file config (83ms)
    1) Show the message create file config 2


  1 passing (6s)
  1 failing

  1) Check configuration without modules Show the message create file config 2:
     Error: An element could not be located on the page using the given search parameters ("#module_2_helloworld .module-content").
      at elements("#module_2_helloworld .module-content") - getText.js:18:17
      at execute(, "isLoading", , "require") - api.js:350:19

I pushed a sample for shared with you and these are passing on Travis
https://travis-ci.org/roramirez/MagicMirror/jobs/200521954

Anybody can give me some light?

qistoph

Weird… don’t see anything wrong with your code and it also runs fine on my machine…

Probably not really helpful, though, sorry

roramirez

@qistoph I’ll test in other instance.

What S.O are you using?

qistoph

@roramirez what do you mean by “S.O”?

roramirez

@qistoph Sorry, Spanish acronymous. I mean Operating System.

On the another hand. The same commit is passed in Travis on my personal forked but in the main Mich’s repository failed.

https://travis-ci.org/roramirez/MagicMirror/builds/200750569 (Pass)
https://travis-ci.org/MichMich/MagicMirror/builds/200750793 (Fail)

It’s so weird.

qistoph

@roramirez my tests ran on Ubuntu.

The failed build you’re linking has timeouts. That’s the issue I added to the Trello board. We could increase the timeout a bit and see if it happens again in the next days.

Ask Mich to re-run the failed build, he should be able to click that option in Travis, being the owner of the project. See if that fails again. That resolved it for me before.

roramirez

@qistoph it’s pushed https://github.com/MichMich/MagicMirror/pull/716

roramirez

Hey guys,

I pushed up a small refactoring for tests.
https://github.com/MichMich/MagicMirror/pull/728

This is a improvement for create new tests without unnecessary repeat inside of code.

Something I wasnt think now is how can be integrate the case test created by @qistoph because this is set a app.path inside of test (It’s something I have not tried…yet)

https://github.com/MichMich/MagicMirror/blob/e62df3b3b18cc98b65656194fa5baa53b9110fbc/tests/e2e/dev_console.js#L37