How to design testing library?

Question

I would like to create a sort of testing library or framework. The main purpose of it is to run UI tests on Android device. For example, here is example of how I would like it look like:

(test-flow
    (launch-app "some.app.id")
    (assert-visible "Login")
    (type-text "admin")
    (tap-on "unlockButton")
    (assert-visible "HomeScreen"))

I hope this API looks nice and clear. Basically here is what it is supposed to do:
- launch application with id "some.app.id"
- check that login field with name "Login" is visible on the screen
- type a text "admin" (into login field)
- tap on unlock button
- check that text "HomeScreen" is visible, what means we successfully logged in.

I've already created all the desired functions above and that was easy part. I've designed them so they return a hash-map that may have either :result or :error. But I've been stuck on how to use those functions together, so it will work as a testing framework. If I put them inside some function and if some of the steps fails, then execution will just continue to the next step. Moreover, the result of a failed step will be lost. After that I decided to write a macros, that will transform all functions into list, evaluate them one by one and return detailed result for each step. But I failed to do that (event with the help of ChatGPT), maybe because I'm noob in Clojure.

After probably a week of inability to solve this problem, I think maybe I'm doing something wrong. Maybe someone could help me with that API problem?

Here is my vision of that API:
- the result of the test should have results from all executed steps (functions)
- if one step fails, then execution should be stopped, and results for all executed steps should be returned.

I was inspired by this framework, I just think it would be much easier to write such tests with REPL and do some Clojure magic inside the test :)

Thanks!

Comments

Sounds to me like you have a design problem and that is difficult to answer directly because the best answer for you depends on how you expect this "testing library" to work.

Two ideas that might help:

1) Have a look at how `clojure.test` is implemented. There might be some ideas there that will help. https://github.com/clojure/clojure/blob/clojure-1.11.1/src/clj/clojure/test.clj - only 800 lines. Lines 706+ are the ones dealing with runtime execution of tests.

2) I suggest treating your test flow as a data structure:

```
(def test-flow
    [[launch-app "some.app.id]
     [assert-visible "Login"]
     [type-text "admin"]
     [tap-on "unlockButton"]
     [assert-visible "HomeScreen"]])
```

then writing functions that operate on that to execute it in a controlled way and monitor the results. Then once you have the data structure that works for you, write a macro to handle the quoting if that is needed. Focus on something that is easy to debug.