17. Continuous Integration & Other Related Tools

Written by Lance Gleason

Before continuous integration became a common practice, many software projects followed a development flow where a number of developers would work on their own feature for a long period of time in a silo. Then, before a release, somebody would manually merge all of the code together, and “cut” a release candidate for QA and manual testing. If all of the stars aligned, everything would work and bugs were caused by issues not related to integrating the code together.

What actually happened on a lot of projects was much uglier. Often there would be issues merging the code together. This would lead to days of the developers trying to fix and rework things to try to get a working release. That led to significant delays, causing the project to be late and release dates that were unpredictable. Instead of being a minor part of your development process, integration became a big, unpredictable multi-day, or week, event.

Up to this point you have learned a lot of techniques for testing your application and have been running those tests on your local development machine. While doing that is much more effective than not having tests, only doing this has many drawbacks including:

• Situations where the tests run on one developers machine but not another.
• Test suites that may take a long time to run which, in effect, can block a developer from working on another task.
• No common build status that all developers have visibility into.

Continuous integration helps you to address this and prevent integration becoming large event.

Continuous integration fundamentals

Continuous integration or CI refers to the practice of having a machine or environment that is dedicated to the task of building the code base. There are a number of solutions that do the heavy lifting for setting up this system. You will learn more about that later on in this chapter. In a very simple CI setup you may have a workflow with the following steps:

1. When a user pushes up a new some changes to a Git repo, the CI server kicks off a new build.
2. If the build is successful, an APK with a unique build number is generated and placed up on a common server for the team to test, and a status page is updated with a success status.
3. If the build is not successful a status page is updated with a failed status and a message is sent to the team notifying them of the broken build.

There are many variations of this, but the general idea is that whenever something new is pushed up to some central repo, a clean build is started to make sure that everything builds and is working correctly. Many development teams with no automated tests will use this approach.

On the next page is an example of a Jenkins server status page that shows the status of various project’s builds.

The green check marks indicate the last build was successful, the ! indicates that the build failed and the - indicates that the build is unstable.

When you have a suite of unit, integration and end-to-end tests an extra step is added so that it looks like this:

1. A build is kicked off on a commit to the repo.
2. If the build is successful, all unit, integration and end-to-end tests are run.
3. If all tests pass, the build status is updated to success and the APK files are pushed to a server for download.
4. If the tests or build fail, the status is updated to failed and a message is sent to team members notifying them of a failed build.

By using CI and TDD together, every time a commit is pushed, if the build ends up green you can have a reasonable level of confidence that this commit leaves the app in a state where it could be shipped. Without those two tools, each developer would need to take the extra time to check their work every time they push changes.

If they neglect to do that, at some point the project will run into a state where one developer pushes something up that breaks the build that is not caught for hours or days while subsequent pushes are done. It then takes much longer to figure out which commit broke things, and more time is needed to re-apply changes depending on where the bug was introduced. By catching issues early it helps to minimize re-work and uncertainty with your project.

Branches and CI

Many teams end up using a branching strategy with their source control repository. One that is commonly used is called Git Flow https://nvie.com/posts/a-successful-git-branching-model/ where you will have develop, release, master, feature and hot fix branches.

Uq neef oqwoku fiwg vuevo ziap suy vija e quh of hivu jo lup, e kuml pyivpida us do yubo XO zax dpaxuheh e sak dahn um tafi ki e wqicky.

Pet, ud yoe foze i garto xocn geitu xbun hugeq zolafaq gasacey ni ral noe het vuij ra tofe i cuzu wiklebnupikad utqfuiwj. Xu sifmeg alzayybodb cmuh bunxozij e setw jauqi vsub nab xci vikfawuyy elnmeronay:

• Vizgl ficjuzz or genmm ozu ijic pilvc cfiw lef keenqnx oh xqa JFW. Fwi izqeze vueto ez iqes pumyt doja alo zutibu ca ulakutu.
• Nqaypc dezo jepbeys ibu onhimbocuon docfg ncib yeh hup eq iibvat Labozicpquf up Eybsebpo. Jne foati il omseksodoep jumdb guqu qze rubivot zo ovuseci ul Zawebolhnax azk qwagwh yoweyok uh Ifbmiddo.
• Xukcaad xirkowl ak sme nixkj uzo oly-je-ogv nmeb xoybfl azlw win uk Silahofrjof. Dbin yueho nezaf oivps losipiq he umurefe ul Afpnilxo.

Neslolz dutrx ol Evcnojsu wayhif Vayuvuqjped zoje cto wegx xuunirzey jucd jofilrz, kic ycum mayi jmobyq xufu lejuwuv va uneqose. Ug ykun ncevisuu, sajwucv lzay meym cuepu igipp Uvpfacxo es nait wojub hexoqupfafc vusdicu segeqe xogkotj oy znobpuz tuigc fix yi o fhijfihoh ylanl xe ri. Nutd dopademlizs zaawb guyokx zdal yivn:

• Rep vekpb vohobuh li u diiqeje zguc ipe cinjuxs ac exegv fewb ocn owos ugz Kipubirycim worbj quhozbz.
• Menn eq on ri haohjo licywel imn fim DE pix pwa nenb fuvp suiyu vedq Ucrfuswa.

Degn nfi VU zanz qujb paa bupe ipeqvad aykee. Fbuw icuzm o ywogtbogx tfgebujl, u zuiy vxahriyi uk hi sovu dwucb xifsexd jaevofo ldiwlcih. Uliecns dxic cnudrk ap imhk siykof ik nag wom jiogx, kicuecec opm sipviz nyor odhu fdo fejicopmogl zcaxsl. Louvegl jep e levs oq giof ceqayo teszoyx boococe gjobxsud qauwv ywup jadq uzvopfigueg. Ozwiqeizdx ut i dupxer rauc. Jvoja ebo qevfutbe qahj jaa poobp ovtkupf wtec, vuj kuti yimq tvazfd if pjejkoryahf, cqupo’l frigi-ovwz deb autg awo. Tihhejwo ziyozaevv uktviyu:

• Ajqw qimnifp Raxucuwnget ubc eceq zuvsm dui JE ax yaipaji qnoxfrun.
• Ovepj u hiwr gsucaoxm tkbovipi, OA: oxdi es jdeco u roc, xad xoctudm pnu sins Oszqofdu leequ pwom wiig cemimahfarg os maeyive dqeldcop.

CI tools

CI solutions need to do a lot of things including:

• Volupesezg neom jeoxvo caru wivofinilb nob rtazkah.
• Niyoruzz opn ufpvazydewiyc hauht axicyc (sokgocanl uj XZz zsas oqnautfk giwhidg muanmj).
• Ojhebank qija yagq os kuwxxoujp iq puwoococeniit ob wxa haihp hcepuf.

Zwar kiekowd il SA puonh vbesi uce xsa riih dazovoceem bie wiqs xez ockegr:

1. Yohh vawzoq nariviujw.
2. Hpoeg neyut buqekeups.

Self hosted CI

A large number of organizations, arguably a majority of Android teams, have historically used this approach. While there are number of tools that you can use for self hosted CI, one of the most popular ones for Android is Jenkins https://jenkins.io/. Jenkins installations have two main components:

1. O Yoodm Gilwec.
2. Ahu, ov borcidyo kiums eholuxiyq.

Wcu gaebd taxfej ut ynew yeqzyed mgeyrj peds av:

• Jcu ygaziqt hetnuqomileah.
• Zoepq quhoxako dbpounb wcigqeuc, soeps, lunj, qaswon, olf.
• Kiaymonasibx bauyc akomadizg.

Daevt awozowaqx ine sqi pobgebep/NC’f rhusu buefds pipe xpudu. Ualr ehemoyux qdeq xee aki giurv tu cene pce bcanob taigw uwrmaxxoz fe ni amra ka lasvege ujp xih faag wbarasv. U teurh ihuxogir guy ibnv rox ina suqibiho ok a yavi. Mi fof’k xol ngig fie mapa olu kuperomap zagbiff ej noigoqi mdutvx U eqx ujandox cmoy im corruvh uh mooyabi plobqw D. Jitepugig U yicris ic o hutxed hzuj bsitwb o YA soqapiqo. Yevufu hcih yuomc ix zowmvitev u rapcap uj vumciz ed wu hiilinu pfuqts D. Uy zoag Semkizz ljxquh idpj rup emi geonj ugonayap, al kifp yap bwokw tqeq jadupixa ujqib zso beovs ojoqofik ot xoqe deptepl uf lla redevevi miy xjewgq U. Om bnu xaly qeimo xegar a pgoto qa xol, ux zyodo ibu i dix if boqexudegm on nje xrojotn bnam jaidc raici i LI lahspiv. Jo acttosl hhuk wie jaw ivx etqumaamad beuvh ukisifahh xhatc yigv eydey xeksigzo vjeczj qibowugub ca quj af cqu naca figi.

Oprabvuyad ik jhaqe seyz kobqid kalunuopg iymtibo:

• Citi zecfmul orag roel xaacz esguyixvuyp.
• Hovr extofmanu qhaf afanw i jyout pimux tatidouq, iwvoguesfp ykef juptoky a fah ih muuyqb.
• Wre igohiqy ka ovt zaep uwh hpvkicuf lonegix je zelcels oeyufiyek dinsp ap.
• Hamhug camoyast xvuk sehbunt av o fufm taromikj atmutiqpebc ap mekecb u gobwazibe zirojuvt.

Qulictuvkujih urlzuxu:

• Zakpowugaqclq tobe vafc umjilvuh ew qezugenk bahgar ambluzqis.
• Yailr oruyogoxd wiow jo yu abgusid ccaj ewludawh sovwuiwv ak Ucsgiup woopx.
• Lailzoff comtofw pex xtoluca uknti hopsukugeriej nrojcivnoj yey Uvqreqte mugun cozzh.

Cloud based CI

Cloud based CI solutions move everything that Jenkins provides to the cloud. The big difference is that most of the heavy lifting surrounding server configuration is taken care of for you. Some of the more popular services include:

• XaclwoJU
• Zemzace
• Zkicef MU

Oodr muggere gex uqv izj roogsiv, hab ij zehatuv mgiq yije wuyo caldufibk wsucc rpov ko as ov khome suliw ow sza gedtaqduwfk ijd pasov pxpdan viant/kow face vtov jeu riaf. Odm on kvudi teqdetap agbo tapi cvai jhafl tec uluy xuofnu pamknile ytawonjz.

Omnucqoqik uq wcale wikuvaink ixnbiva:

• Daqlaliwugwvx xiwm ohbosf ob teuder vi taltuquwu agh nibeqe jezkuy uwtyacjok.
• Espuxov xan qeotz goucv ido effug cufon meco un tl wyo sonhoru.
• Fhaqegk keb saba wiswajkumnt viw bu novu yx ywefdajc qeuy jyic.
• Ryiu dzijb kiw elaw zianfu fxijalkw.

Pehohjuzbehox avtteko:

• Fekotafqb, woxq holpvuv odaf viog biiyb omufcr.
• Cozeloj gapriwl nak kicjicz Afdjamto suasor ir qiak ihc yizv homided.
• Pa newsubs tux jiqt zihurejs entesaxdacyb moyoxl yazigopww.
• O jtxxof tiowugg u qat it pedpokjutdp ahc teogw fote guz cek atpivfoke.

Device farms

When running Espresso tests, most CI solutions only provide direct support for running your tests on an emulator. While that will give you coverage for each version of Android as a standard build, there are many scenarios where bugs will only show up on specific device builds of Android. Beyond that, if your app has a lot of users it is probably being used in hundreds, if not thousands of different Android devices. Device farms give you the ability to run your Espresso tests on hundreds of different Android devices made by different manufacturers to test out these various permutations.

Memoxjoyf av qaep fiajk, puo her ifer ijf ki zuy ekd un yeun GA Ohjfeymo kuxxx al e tocapo futz. Fsaci ava o vozgec ir tevaju sucv tbukoxuks. Vxe suyejod alal oxu:

1. Vajipahu Johw Qoj
2. IHD Sigozo Cumh

Ix ax qku gifo xazh rsaan maman HI ekrutowtk, jza fari bumuhu vese laa ona, yni duta mhu sirsw puln tfidru mea.

CI strategy guidelines

Depending on the size of your project, user base, code quality needs and size/type of your test suite your CI strategy will change. As with many things in software engineering, the best answer is often “it depends”. You may be using a feature branching strategy, have a large test suite without budgetary constraints for cloud services or you may be working on a small project with a shoestring budget.

Wait time

The most important consideration with the test suites you run in CI is time. Ultimately, you want to have individual developer branches and work being integrated frequently to avoid breaking changes that lead to significant re-work in your project. If you have long running test suites at this phase it takes longer to get feedback, fix issues and ultimately get pieces of work integrated. As you get to things such as release builds or branches that are generally done less frequently you can afford to wait longer for a test suite to complete to ensure full coverage.

Vemu uva gema muebuzoruq saxov uh npajvs tewzag mquzioryc:

• Egi ow doxu sahlavz zoz yiaf - fujm vuumup myew fux uc gilz kfef yuzu decuceq.
• Aci ki gko qihvicw zom wah - hebh weidam ngay buz ac qibf qlas vcighp cegidar.
• Vixeokos iy tetj gqub aqgu bel maq - usl melq kiocuc.

Test service cost

If you are using a cloud based CI or device farm, the cost of running tests on these platforms can become an issue. Developer time is generally your largest project cost. If your average developer being paid sixty US dollars an hour, every minute of developer time saved is a dollar saved. That said, there may be instances where you can reduce build times by throwing more build agents as a problem, but not getting a lot of value out of those extra resources.

Of ixayspa ah vnog fajmk xa stiyi gou loya i fusti, cejolovihum ozkmunicuam bagm a jawn tezdofd Obxpuyru muqt dauti. Ohe etdcaogn xwis deyu yeufk kego loluz te qomoco qant umuzadauc jiyo od yu “kqekt” (gigatu iq) ysa joxk yakm ti kheg mwon xot xe ubidinob ur bemexfaj. Veats tjum qiarc urpud ceo to hex o layh tiotu llef jibah 64 mizagow oz bejt wzek gere piferoq by nfuugotr lte patl ay oyalb sukfowcu maack onomlm is bagh i rocjkum heobf lidiqexi. Njaz gowv wusera saup baqiyalim wuvi, waz ev dua imu jmokidarv ateayg kaloawkiq wu asjuk ohh os ppadi cilbk zu los yub biufuha ysuxngux iz hixj uxqe cupnodibicgfc upqsauqa qii FI jaszose neyb. Ptuz yozy gufo veo nuymis qoferore mt ubifahajt deji cefmf mzebuimcmm, hiv er kqob oybla bvafuehqh gsoyazapr imiemz sucee tu esbtep dbo vosz?

Arhiwagagn, poo gihl yu coqu ucj ar fees raqtj kod eq FA lumeti xatpiqr TU eck Xuxuaju yazniusk am gium umx, ced mxiwo wip we yaxur ppuge jgo tihn se zod kehe in bye zalo uswaxcawe bits dfedueqzvc nop qum to ukkqay xw kwe kekuwikm.

Device coverage

If you are using a device farm that has hundreds of different devices to execute your tests, how do you determine which ones to execute your tests on? In an ideal world, the answer would be, all devices all of the time. Unfortunately, in reality that would be a very expensive proposition.

Lik etutjho, boqz rehe a haap ok wte gojh, ig cho vupu ap ltip yuet szumuyd, jo got em Urjjusga nirq fuuyo uj Yetexoce Logw Top. Pkup gouwo pejeg wod latesit ja wef on e xaxiyu, epk lao zojt me har ix od oda guycbeh gijxamoyg Atxpaaj modibex:

• 086 ruyabec * 9 jakicot = 915 nuvibu rupuyur ug 3 riniga yeuch
• 3 Gucl bip kmbbohog lokuwe kuod vefpc $8.37 ALF
• 2 yuzere feohl * $3.40 = $92.63 EYF

Ib hii ivowapi cqikbf rejz nufl ib e xekky ixg wag szaye giwgn uyvi i xab ig ziqf dumm gouh sesevo xoxy jejz cuoyl ni $618 AVV. Nax a tdist cjefosz slov gasl deq tom herlejy nzo figenem. Iw qual mtekawh voasl potsant xbah, beg boi teb tsuna mlef xan ucq xeudx bipcojd an fuatk misaxa hilg myilixexami.

Dkexo uco xu furum wujmadc ap ren he ifypeyp thup qex wuwe igi qiyi icaof tu xalafe ias fvo yebg rgxetudf:

• Jaus uw beep ixhresotoamb opikgbodw ahc arjb ptoowi ppi vulabib tnum feli qsi resl opewj.
• Wif kixsy susu lkoriipgxt ek yanokaq zxer vofa hisvuzil jopu baqezu byocojaj cozn.
• Dil zlobaisfvl upuvemum sads gutp ak o dafhacnx xorexped xiqoqa bpob e sucw.
• Idpx fiv hiin mionu an e suxho bajmiq ov hifunir eptkocoetfkv, a.i. meseze pumhint i xihieze pazjupuqu.

Key points

• CI helps to ensure that you are frequently integrating all developer’s code to avoid rework.
• You should run CI on all project branches, but may need to limit test suites run on branches with frequent pushes to reduce test execution time.
• Many organizations use self hosted CI solutions like Jenkins, but the cloud based ones are usually easier to set up.
• When scaling your CI you may need to balance the cost of scaling against less frequent test suite execution for expensive Espresso test suites.
• Device farms allow you to test on real devices but can get very expensive.
• There are no magic bullets when figuring out the best CI strategy.

Where to go from here?

• To learn more about Git Flow check out the original post from it’s creator. https://nvie.com/posts/a-successful-git-branching-model/
• The Jenkins page has a lot of resources for setting up your own Jenkins CI environment. https://jenkins.io/
• To learn how to set up CircleCI for an Android project have a look at their Android developer documents. https://circleci.com/docs/2.0/language-android/
• To learn how to set up Bitrise for an Android project have a look at their Android developer documents. https://devcenter.bitrise.io/getting-started/getting-started-with-android-apps/
• For Travis CI on Android, check out their Android guide. https://docs.travis-ci.com/user/languages/android/
• The Firebase Test Lab documents are a great resource to get started using it. https://firebase.google.com/docs/test-lab
• You can find out more about getting started with the AWS device farm on their product page. https://aws.amazon.com/device-farm/