Abstract
We present a method to construct a semantic map of an apple orchard using a LIDAR and a camera rigidly attached to each other. The system is able to capture the map as a standalone sensor which is light-weight and can be mounted on a variety of platforms. At the geometry level, we present a new method to associate image features captured by the camera with 3D points captured by the LIDAR. We then use this method to register 3D point-clouds onto a common frame. We show that our association method yields superior registration performance compared to common methods which work in indoor or urban settings. At the semantic level, the apples are identified as distinct objects. Their locations and diameters are extracted as relevant attributes. As an example, a semantic map of an orchard row is constructed.
| Original language | English (US) |
|---|---|
| Pages (from-to) | 85-89 |
| Number of pages | 5 |
| Journal | IFAC-PapersOnLine |
| Volume | 49 |
| Issue number | 16 |
| DOIs | |
| State | Published - 2016 |
Bibliographical note
Funding Information:∗Computer Science & Engineering, University of Minnesota, ∗Computer Science & Engineering, University of Minnesota, Computer Science & Engineering, University of Minnesota, HorMticnunlteuarpaolliSs,ciMenNce5, 5U4n5i5ve{rpseintyg0o1f7M5,ipnrnoeys,iostlae,r}S@t cPsa.uuml,nM.eNdu55108 ∗∗ HorMtiicnunlteuarpaolliSs,ciMenNce5, 5U4n5i5ve{rpseintyg0o1f7M5,ipnrnoeys,iostlae,r}S@t cPsa.uuml,nM.eNdu55108 Horticultural Science, Ulunbiyvxe0r0si1t@y uomf Mn.iendnuesota, St Paul, MN 55108 Horticultural Science, Ulunbiyvxe0r0si1t@y uomf Mn.iendnuesota, St Paul, MN 55108 lubyx001@umn.edu lubyx001@umn.edu Abstract: We present a method to construct a semantic map of an apple orchard using a Abstract: We present a method to construct a semantic map of an apple orchard using a fAlIbDsAtrRacatn:dWa ecapmreesreanrtigaidmlyeathttoadchteodctoonesatrcuhcottahesre.mTahnetiscysmteamp iosfaabnle atpopclaepoturcrheatrhde umsianpgaas Abstract: We present a method to construct a semantic map of an apple orchard using a aflIsDtaAnRdaalonndeasecnasmoerrwahricghidilsylaigtthatc-wheedightot eaancdhcoatnhebre. Tmhoeunstyesdteomnias avabrleietyo coafpptluartefotrhmes.map as flIDAR and a camera rigidly attached to each other. The system is able to capture the map as Aa tsttahnedagleoonmeesternysolervwelh, iwche ipsrleisgehntt-waeingehwt amnedthcaond btoe masosoucnitaetde oimn aagveafreiaettyuroefs pclaaptftourrmeds.by the a standalone sensor which is light-weight and can be mounted on a variety of platforms. cAatmtehreagweiotmhe3tDryploeivnetls, cwaeptpurreesdenbtyathneewflImDAetRho.dWteotahsesnocuisaetethiimsamgeethfeoadtutroesrecgaisptteurre3dDbpyoitnhte-At the geometry level, we present a new method to associate image features captured by the cclaomuedrsaowntitoha3cDompmoinotnsfcraapmteu.rWedebsyhothwetfhlIaDtAouRr. aWsseocthiaetniounsme tehthisomd eytihelodds tsoupreegriiostrerre3gDistrpaotiinotn-cclaomuedrsaowntitoha3cDompmoinotnsfcraapmteu.rWedebsyhothwetfhlIaDtAouRr. aWsseocthiaetniounsme tehthisomd eytihelodds tsoupreegriiostrerre3gDistrpaotiinotn-pcleorufodrsmoanntoceaccoommpmaroendftroamcoem. Wmeonshmoweththoadtsowuhricahsswocoiraktioinimndeothoordoryiuerldbsansuspeettriinorgsr.egistration pcleorufodrsmoanntoceaccoommpmaroendftroamcoem. Wmeonshmoweththoadtsowuhricahsswocoiraktioinnimndeothoordoryiuerldbsansuspeettriinorgsr.egistration Apetrtfohremseamncaenctiocmlepvaerle,dthteoacpopmlemsoanremideetnhtoidfisedwahsicdhiswtionrckt ionbjinecdtoso. rTohreiurrlboacnatsioenttsinagnsd. diameters Apetrtfohremseamncaenctiocmlepvaerle,dthteoacpopmlemsoanremideetnhtoidfisedwahsicdhiswtionrckt ionbjinecdtoso. rTohreiurrlboacnatsioenttsinagnsd. diameters aArtethexetsreamctaendticaslerveelle,vtahnetaaptptlreisbuarteesi.deAnstifaiendeaxsadmisptlien,ctaosbejmecatns.tiTchmeiarplocoaftaionnsoarcnhdadrdiamroewteriss aArtethexetsreamctaendticaslerveelle,vtahnetaaptptlreisbuarteesi.deAnstifaiendeaxsadmisptlien,ctaosbejmecatns.tiTchmeiarplocoaftaionnsoarcnhdadrdiamroewteris caorensetxrutrcatecdte.d as relevant attributes. As an example, a semantic map of an orchard row is caorensetxrutrcatecdte.d as relevant attributes. As an example, a semantic map of an orchard row is constructed. c©on2s0t1r6u, cIFteAdC. (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: computer vision; agriculture; image reconstruction; image recognition; object Keywords: computer vision; agriculture; image reconstruction; image recognition; object rKeecyowgnoirtdiso:ncomputer vision; agriculture; image reconstruction; image recognition; object rKeecyowgnoirtdiso:ncomputer vision; agriculture; image reconstruction; image recognition; object recognition recognition 1. INTRODUCTION 1. INTRODUCTION 1. INTRODUCTION Robotics and sens1.ingINteTcRhOnoDloUCTIOgies areNfinding increasing Robotics and sensing technologies are finding increasing aRpopbloictaictsioannidn asegnriscinugltutreaclhsncoileongciees anrde pfirnadcitnicgesin. cInreathsiensge Robotics and sensing technologies are finding increasing applicationnsi,n daagtraicuclotullreacltesdciebnycesseansdorpsraicsticteusr.nIend thinetsoe application in agricultural sciences and practices. In these ianpfpolrimcaattiioonns,atdavtaariocuosllelcetveedls.bAytsaenvseorrys bisasitcurlnevedel,inthtoe applications, data collected by sensors is turned into sinefnosromr adtaiotna aactqvuairreiodusfrolemveals.mAotbiale vpelraytfobramsicislemvealp, ptehde information at various levels. At a very basic level, the dseirnescotrlydattao arceqleuviarendt firnofmormaamtioonb.ileAptlaytpfiocraml eixsammapplepeids dseirnescotrlydattao arceqleuviarendt firnofmormaamtioonb.ileAptlaytpfiocraml eixsammapplepeids tdhireecmtlayppitnogreolfevaaenrtialinifmoramgeastiotno. NAortmyapliiczaeldeDxaifmfeprleencies tdhireecmtlayppitnogreolfevaaenrtialinifmoramgeastiotno. NAortmyapliiczaeldeDxaifmfeprleencies VtheegemtaatpiopninIgndoefxa(eNriDalViIm) atgaebsletsobaNsoerdmoanlizmedultDi-isfpfeercetnrcael the mapping of aerial images to Normalized Difference cVoelgoertaatniodn iInntdenexsit(yNDdaVtIa). tAabtletshebanseedxtonlevmelu,ltgi-esopmecettrraicl Vegetation Index (NDVI) tables based on multi-spectral mcoaloprs aarnedbuinilttenassitayn dinattear.mAedtiattheestneepx.tFrloevmelt,hgeseeo meatprisc, mcoaloprs aarnedbuinilttenassitayn dinattear.mAedtiattheestneepx.tFrloevmelt,hgeseeo meatprisc, imnafoprsmaarteiobnusilutcahsaasncrinotperhmeiegdhitatceanstbepe.dFireocmtlytheessteimmaatepds, maps are built as an intermediate step. From these maps, finorforemxaamtiponlesfuocrh pahsecnrootpyphienigghtapcapnlicbaetidoinrsec(tSlyeeesftlimeatteadl,. 2. REflATED WORK finorforemxaamtiponlesfuocrh pahsecnrootpyphienigghtapcapnlicbaetidoinrsec(tSlyeeesftlimeatteadl,. 2. REflATED WORK (fo2r01e4x)afmorplaenfoovrerpvhieenwo)t.yping applications (See fli et al. 2. REflATEDWORK (fo2r01e4x)afmorplaenfoovrerpvhieenwo)t.yping applications (See fli et al. 2. REflATEDWORK (II((nn2014)2014)20oo1rder4d)efffrrororortotoanananturntuooorvvvneeerrrssvvveensniiiew)ew)eswoorr)... dadatata iinnttoo aacctiotionanableble infoinformrmaatiotion,n, Semantic mapping can be very useful in agricultural ap- wInhaotrdiesrutlotimtuartnelsyennseoerdeddatias iant”oseamctainontiacbmleaipn”foirnmwathioicnh, Semantic mapping can be very useful in agricultural ap-In order to turn sensor data into actionable information, plimcaatniotnics mbyappprinovgidciang breelvevearyntuisneffourlminataiognrictuoltfuarraml earps-, rwehleavtanistuolbtijmecattsel(yplnaenetds,edfruisitas,”bsreamnacnhteisc, metacp.)”ainndwthhiecihr plications by providing relevant information to farmers, rwehleavtanistuolbtijmecattsel(yplnaenetds,edfruisitas,”bsreamnacnhteisc, metacp.)”ainndwthhiecihr it is important to locate individual trees and fruits, and relevant aotbtjreicbtuste(spl(asniztes,infrutihtrse,ebdraimncehnessio, nest,c.c)olaonrdactrhoesisr estimate their sizes so as to calculate useful yield param-mreulelvtiapnlte aspttercibtruat,eest(cs.)izheaivnetbhereene iddiemnetnifsieiodn.sI,nctohloisr paacproesrs, it is important to locate individual trees and fruits, and relevant attributes (size in three dimensions, color across itstiismiamtepotrhteainrtsitzoesloscoaates tinodcivalicduulaaltetrueseesfualndyieflrduiptsa,raamnd-wmeulptirpelseenstpeactsryas,teemtc.)whhaicvhe cbaenenaicdqeunitriefiebdo.thInathgiesompaeptreirc, estimate their sizes so as to calculate useful yield param-wmeulptirpelseenstpeactsryas,teemtc.)whhaicvhe cbaenenaicdqeunitriefiebdo.thInathgiesompaeptreirc, etxetresn.siTvehleytionpitcheofcsoenmteaxnttiocfmaopbpiilnegrohbaostsbe(eKnossttuavdeielidsesetirms.aTtehtehetiorpsicizeosfssoemasantoticamlcauplaptiengusheafuslbyeieelndsptaurdaimed- mweappraensdenat saemsyasntteimc mwahpicohf caannoarchqaurirdeubsointgh oanglyeocmameterriac eters. The topic of semantic mapping has been studied mweap raensdenat saemsyasntteimc mwahpicohf caannoarcchqaurirdeubsointgh oanglyeocmameterriac exntdenGsaivsetelyratinost(h2e01c5o)n).teInxtthoifs pmaopbeirl,ewreorbeosttsric(tKoousrtsaevlevleiss map and a semantic map of an orchard using only camera exntdenGsaivsetelyratinos t(h2e01c5o)n).teInxtthoifs pmaopbeirl,ewreorbeosttsric(tKoousrtsaevlevleiss GanPdSfloIrDiAneRrtiimalasgensso(wrsi)t.houtadditionalinformationfrom aanonddwGGoraastkstdeeirratraetcoostsly((22015)r0e1le5v)a))..nIIntntttohhaiissgppriaapcpueerltru,, rwweee. rrWeestsetirrsiisccttaoounudrrsselBeilbvveessr GanPdSfloIrDiAneRrtiimalasgeensso(wrsi)t.hout additional information from (2010) presented a semantic place classification system for GPS or inertial sensors). to work directly relevant to agriculture. Weiss and Biber OGuPrSmoraiinnetretcihanl isceanlscoorns)t.ribution for the geometric level is outdoor agricultural robots. In contrast to our approach Our main technical contribution for the geometric level is o2u0t1d0o)orpraegsreinctueldtuaraslemroabnottics.pIlnacceocnltarsassiftictaotioounrsyasptpermoafcohr OuraOunrovmmelaainimn etettehccohhdnicnicfoaarll ccaoocnnctturrriiabbtuuettliiyoonnasfofsoorrcthetiahteingggeeoofmmlIDeetrictAriRc lelpevvoeeinll tisiss outdoy usoredaghricighulturaresolutiol robnoRtsT. K-InGPScontderasvict toes foourrcaoppnstrucroact-h aOunrovmelaimn ettehcohdnicfoarl caocncturriabtuetliyonasfsoorctiahteinggeofmlIDetAriRc lpevoeinl tiss ohuetydouosredaghriigchulrteusroalul trioobnoRtsT. KIn-GcPonStdraesvticteos foourrcaopnpstrrouacth- wa inthovieml amgeetfheoatdurfoers wachciuchraatlelloywassfsoorctiahteinqguifclkIDreAgRistpraotiinotns ing the maps. Zhang et al. (2014) presented a landmark-wa inthovieml amgeetfheoatdurfoers wachciuchraatlelloywassfsoorctiahteinqguifclkIDreAgRistpraotiinotns inhgeythuesemd ahpigsh. ZrehsaonlugteiotnaRl.T(K20-1G4P) Sprdeesveincteesdfoar lcaonndsmtraurckt-- wowfithitfhlIimiDmAaaRggee pfefeoaainturetutrecsslowwudhhsiicchhthaaalltlloowwcassnfofobrrethethueseqqduuiitccokk rerreeggcisiosntratrsatrtiotuiocnnt ingasedthememtahops.d fZorhacrngeatetinagl.a(2lo0c1al4)mappresofenttehde eanlavindmronmaerk-nt owfitfhlIiDmAaRge pfeoaintutrecslowudhsichthaaltlowcasnfobrethueseqduitcok regciosntrsatrtuiocnt bnagsetdhemmetahposd. fZohracnrgeaettinagl.a(l2o0c1a4l)mparpesoefnttehde eanlvainrodnmmaernkt- tohfeflIpDlaAnRt pstoaitnutrec,loushdaspteh,atancdancabneoupsyeddetnosirteyc.onRstarpuicdt, using a flIDAR. Their system is mounted on a ground tohfeflIpDlaAnRt pstoaitnutrec,loushdaspteh,atancdancabneoupsyeddetnosirteyc.onRstarpuicdt, based method for creating a local map of the environment ethffeiciepnlatnthrsetae-tduirme,ensshiaopnae,l apnladntcacnaonpoypydernecsoitnys.trRucatpiiodn, vehicle. They use the vehicle position determined by en-the plant stature, shape, and canopy density. Rapid, using a flIDAR. Their system is mounted on a ground hefafiscaiepnptlictahtrieoen-sdifmorepnhsieonnoatlypilnagntin craesneoaprychraencdoninstfrourcmtiionng coder odometry to register the flIDAR points. In contrast, hefafiscaiepnptlictahtrieoen-sdifmorepnhsieonnoatlypilnagntin craesneoaprychraencdoninstfrourcmtiionng coehdiecrleo.dTomheeytruysteotrheegivsteehrictlheepfloIsDitAioRn pdoeitnetrsm. iInnecdonbtyraesnt-, haphlaassnaatpplicpcpalnicoaaptiotyionsnmsafofnoarrgphepehmennenoottyypppriianncggtiniincerersesseeuaacrrhcchhasaannpddruinfoinnfinormrgmainginndg coederutiolidomze theteryimageto regifeatsterurteshe ffrlomIDAouR rpoiflInDts.ARIn -cocamerntrasta, phlaasnatpcpalnicoaptyionmsafnoargpehmenenotyppriancgtiincerseseuacrhchasanpdruinnfinorgmainndg woedeurtiolidzoemtheteryimtoagreegfiesatetrurtehse ffrloIDmAoRurpofliInDtsA. RIn -cocnatmraesrta, ⋆plant canopy management practices such as pruning and system to register the flIDAR points. Das et al. (2015) ⋆plantcanopymanagementpracticessuchaspruningand wyestuetmilizteotrheegisimteargtehefeafltIuDreAsRfropmoinotus.rDflaIDsAetRa-l.c(a2m01er5a) Award 1111638, NSF Award 1317788, USDA Award MIN-98-G02 pyrsetseemntetdoarseegnisstoerrsuthitee fcloInDsAisRtinpgooinf tas.laDsearsraentgaels.c(a2n0n1e5r), ⋆ This work is supported in part by NRI Award 1525045, RI Large system to register the flIDAR points. Das et al. (2015) AwThisarthed w1111638,MornkDisrivsuppeNinSFiortedtiaAtiwveain.rdpart1317788,by NRUISADwAardAw1525045,ard MINR-98-G02I Large pruelstein-stpeddecatrsenaelnsorcoarmsueuriatse,conoantsihistetrinmgaoflfiamlaasegsienrgrancnagemeescanrcaa,nnaernrd, Aandwarthed 1111638,MnDriveNinSFitiaAtiwvea.rd 1317788, USDA Award MIN-98-G02 mruelstein-stpedecatrsaelnscoarmseuriatse,coantshisetrinmgaolfiamlaagsienrgracnagmeesrcaa,nnaenrd, and the MnDrive initiative. multi-spectral cameras, a thermal imaging camera, and and the MnDrive initiative. multi-spectral cameras, a thermal imaging camera, and Copyright © 2016 IFAC 85 C2o40p5y-r8i9g6h3t ©© 22001166, IIFFAACC (International Federation of Automatic Contro8l5) Hosting by Elsevier Ltd. All rights reserved. CCPooepperyy rrriieggvhhiettw ©© u 22n00d11e66r rIIFFesAApCConsibility of International Federation of Automat858ic5 Control. Copyright © 2016 IFAC 85 10.1016/j.ifacol.2016.10.016
Publisher Copyright:
© 2016
Keywords
- agriculture
- computer vision
- image recognition
- image reconstruction
- object recognition