Labeling Data Format¶
EmotionNet, FPENet, and GazeNet uses the same json data format labeled by Nvidia data factory team. These apps in TLT expect data in this Json data format for training and evaluation. For EmotionNet, FPENet, and GazeNet, this data is converted to TFRecords for training. TFRecords help iterate faster through the data. Please refer to the corresponding section for the json data format descriptions.
Json Label Data Format¶
Using the Json Label data format requires data to be organized in a json file with the following struture:
.
{
"filename": "data/001_01_02_200_06.png",
"class": "image",
"annotations": [
{
"class": "FaceBbox",
"tool-version": "1.0",
"Occlusion": 0,
"face_outer_bboxx": 269.0082935424086,
"face_outer_bboxy": 44.33839032556304,
"face_outer_bboxwidth": 182.97858097042064,
"face_outer_bboxheight": 276.28773076003836,
"face_tight_bboxx": 269.211755426433,
"face_tight_bboxy": 147.9049289218409,
"face_tight_bboxwidth": 182.58110482105968,
"face_tight_bboxheight": 172.5088694283426
},
{
"class": "FiducialPoints",
"tool-version": "1.0",
"P1x": 304.8502837500011,
"P1y": 217.10946645000078,
"P2x": 311.0173699500011,
"P2y": 237.15249660000086,
.
.
"P26occluded": true,
"P46occluded": true,
.
.
"P68x": 419.5885050000024,
"P68y": 267.6976650000015,
.
.
"P104x": 429.6,
"P104y": 189.5,
},
{
"class": "eyes"
"tool-version": "1.0",
"l_eyex": 389.1221901922325,
"l_eyey": 197.94528259092206,
"r_eyex": 633.489814294182,
"r_eyey": 10.52527209626886,
"l_status": "open",
"r_status": "occluded",
}
]
}
Here’s a description of the structure:
filenamefield: specifies the path to the images to train on.classfield: category of the labels for the respective section.annotationfield: annotation chunk.
There are three supported chunk in the annotation including FaceBbox, FiducialPoints, and eyes.
FaceBoxchunk: This is a chunk that describe Face Bounding Box labeling information.
Parameter name |
Description |
Type |
Range |
Example |
|---|---|---|---|---|
class |
The class for the annotation chunk |
String |
N/A |
FaceBbox |
|
Version of the labeling tool for this chunk |
Float |
N/A |
|
|
Occlusion state [ 0 = not occlused, 1 = occluded ] |
Integer |
0 or 1 |
|
|
x coordinate of top left corner of outer face bounding box |
Float |
[0, image_width] |
|
|
y coordinate of top left corner of outer face bounding box |
Float |
[0, image_height] |
|
|
Width of the outer face bounding box |
Float |
[0, image_width] |
|
|
Height of the outer face bounding box |
Float |
[0, image_height] |
|
|
x coordinate of top left corner of tight face bounding box |
Float |
[0, image_width] |
|
|
y coordinate of top left corner of outer face bounding box |
Float |
[0, image_height] |
|
|
Width of the outer face bounding box |
Float |
[0, image_width] |
|
|
Height of the outer face bounding box |
Float |
[0, image_height] |
|
FiducialPointchunk: This is a chunk that describes Fiducial Point Labeling information.
Parameter name |
Description |
Type |
Range |
Example |
|---|---|---|---|---|
class |
The class for the annotation chunk |
String |
N/A |
FaceBbox |
|
Version of the labeling tool for this chunk |
Float |
N/A |
|
|
Occlusion status [ 0 = not occlused, 1 = occluded ] |
Integer |
0 or 1 |
|
|
x coordinate of the ith landmarks point |
Float |
[0, image_width] |
|
|
y coordinate of the ith landmarks point |
Float |
[0, image_height] |
|
|
Width of the outer face bounding box |
String |
N/A |
|
eyeschunk: This is a chunk that describes eyes labeling information. This chunk is not required.
Parameter name |
Description |
Type |
Range |
Example |
|---|---|---|---|---|
class |
The class for the annotation chunk |
String |
N/A |
FaceBbox |
|
Version of the labeling tool for this chunk |
Float |
N/A |
|
|
x coordinate of left eye center |
Float |
[0, image_width] |
|
|
y coordinate of left eye center |
Float |
[0, image_height] |
|
|
x coordinate of right eye center |
Float |
[0, image_width] |
|
|
y coordinate of right eye center |
Float |
[0, image_height] |
|
|
Status of the left eye |
String |
open/close/barely open/half open/occluded |
|
|
Status of the right eye |
String |
open/close/barely open/half open/occluded |
|
Here’s an example of a json file with a sample dataset with two image frames:
[
{
"filename": "data/001_01_02_200_06.png",
"class": "image",
"annotations": [
{
"face_outer_bboxy": 44.33839032556304,
"face_outer_bboxx": 269.0082935424086,
"face_tight_bboxx": 269.211755426433,
"face_tight_bboxy": 147.9049289218409,
"tool-version": "1.0",
"face_tight_bboxwidth": 182.58110482105968,
"face_tight_bboxheight": 172.5088694283426,
"face_outer_bboxwidth": 182.97858097042064,
"Occlusionx": 0,
"class": "FaceBbox",
"face_outer_bboxheight": 276.28773076003836
},
{
"P91x": 395.3500000000004,
"P91y": 196.6500000000002,
"P74occluded": true,
"P28x": 436.44144340908053,
"P28y": 174.67157210032852,
"P52y": 252.53100000000143,
"P52x": 428.9925000000024,
"P32y": 236.48449500000103,
"P32x": 416.6063550000018,
"P44x": 427.65443026467267,
"P44y": 186.9615161604129,
"P99x": 425.75,
"P36occluded": true,
"P75x": 428.85,
"P75y": 190.95000000000002,
"P20x": 389.46879000000166,
"P20y": 178.13376000000076,
"P8y": 313.8318038340011,
"P8x": 407.70466707150143,
"P81y": 192.2500000000002,
"P94x": 427.70000000000005,
"P81x": 393.5500000000004,
"P12y": 268.179948238501,
"P12x": 408.69280247400155,
"P65y": 260.04348000000147,
"P65x": 429.0319800000024,
"P84x": 396.8500000000004,
"P84y": 194.4500000000002,
"P93occluded": true,
"P46occluded": true,
"P43y": 193.31428917697824,
"P43x": 421.12354211680173,
"P14occluded": true,
"P92y": 187.5,
"P54occluded": true,
"P53x": 433.50450000000245,
"P53y": 251.9670000000014,
"P45occluded": true,
"P33x": 426.3480450000019,
"P33y": 238.67140500000104,
"P60x": 413.82301500000233,
"P100occluded": true,
"P60y": 272.07148500000153,
"P23y": 174.7903155211989,
"P23x": 428.12940394815394,
"P90y": 194.9000000000002,
"P13x": 399.2067026100015,
"P13y": 257.903340052501,
"P7x": 388.1395861020014,
"P7y": 304.93858521150105,
"P61y": 262.1309850000015,
"P104x": 429.6,
"P104y": 189.5,
"P83y": 193.2500000000002,
"P83x": 395.0000000000004,
"P61x": 404.5783500000023,
"P50y": 254.6756100000014,
"P50x": 414.2206350000023,
"P100x": 424.8,
"P100y": 191.3,
"P34y": 240.46069500000107,
"P34x": 435.9903300000019,
"P18y": 188.2730700000008,
"P18x": 366.50623500000154,
"P25occluded": true,
"P102occluded": true,
"P46x": 436.0852131464696,
"P46y": 191.82999641609848,
"P58y": 275.0536350000016,
"P58x": 429.2307900000024,
"P77x": 306.5418228495726,
"P77y": 258.61884245799524,
"P97occluded": true,
"P99y": 192.9,
"P10y": 293.87146870350114,
"P10x": 434.97720418050164,
"P48occluded": true,
"P26x": 436.0258414360342,
"P26y": 171.99984513074497,
"version": "v1",
"P27occluded": true,
"P86x": 397.8000000000004,
"P86y": 198.45000000000022,
"P73occluded": true,
"P98occluded": true,
"P2y": 237.15249660000086,
"P90x": 393.3500000000004,
"P29y": 203.3826300000009,
"P29x": 433.6046100000019,
"P101y": 188.85000000000002,
"P101x": 425.65000000000003,
"P51x": 423.6641100000023,
"P51y": 252.5881050000014,
"P35x": 436.78557000000194,
"P35y": 239.26783500000104,
"P66x": 433.70401500000247,
"P66y": 268.0952850000015,
"P19x": 378.4348350000016,
"P19y": 181.61293500000076,
"P98y": 193.45000000000002,
"P98x": 427.85,
"P45y": 187.0802595812833,
"P45x": 433.2353710455805,
"P21y": 176.44387500000076,
"P21x": 398.1170250000017,
"P59x": 422.1730350000024,
"P59y": 274.25839500000154,
"P9x": 431.0246625705015,
"P9y": 312.25078719000106,
"P17occluded": true,
"P11x": 422.7243251895016,
"P11y": 281.81621679300105,
"P70y": 195.95000000000002,
"P79occluded": true,
"P95occluded": true,
"P70x": 395.20000000000005,
"P1x": 304.8502837500011,
"P13occluded": true,
"P85y": 196.6500000000002,
"P85x": 398.1000000000004,
"P69y": 196.95000000000002,
"P24x": 433.0572559142747,
"P36y": 236.88211500000105,
"P36x": 427.5409050000019,
"P94occluded": true,
"P104occluded": true,
"P47occluded": true,
"P40x": 401.35650000000186,
"P40y": 197.40000000000092,
"P71x": 396.40000000000003,
"P71y": 196.8,
"P65occluded": true,
"P26occluded": true,
"P56y": 273.06553500000155,
"P56x": 433.0081800000024,
"P16occluded": true,
"P89y": 196.2500000000002,
"P89x": 392.4500000000004,
"P48x": 428.54500592120047,
"P48y": 195.45167075264504,
"P16y": 216.4016531475008,
"P16x": 360.47179483200136,
"P15occluded": true,
"P24y": 170.63429579073562,
"P78x": 276.3975906000002,
"class": "FiducialPoints",
"P74y": 190.10000000000002,
"P4y": 270.1562190435009,
"P4x": 329.2467161130011,
"P96y": 191.10000000000002,
"P74x": 427.85,
"P103y": 195.00000000000003,
"P103x": 396.4500000000001,
"P80x": 330.41417158035716,
"P80y": 178.5832276794402,
"P37x": 381.05250000000177,
"P37y": 200.64300000000094,
"P47y": 195.09544049003392,
"P47x": 433.47285788732125,
"P64x": 432.80937000000245,
"P64y": 255.47085000000143,
"P76y": 191.60000000000002,
"P57y": 271.77327000000156,
"P99occluded": true,
"P43occluded": true,
"P88x": 392.8500000000004,
"P88y": 198.45000000000022,
"P17x": 335.9660368500013,
"P17y": 206.7179262030008,
"P96x": 431.05,
"P67y": 268.3935000000015,
"P27y": 173.42476618118954,
"P27x": 436.38207169864535,
"P87y": 199.45000000000022,
"P87x": 395.1000000000004,
"P3x": 316.76397300000116,
"P67x": 426.8450700000024,
"P96occluded": true,
"P12occluded": true,
"P97x": 430.35,
"P97y": 193.05,
"P101occluded": true,
"P55occluded": true,
"P93x": 429.05,
"P93y": 195.4,
"P42x": 388.6665000000018,
"P42y": 200.64300000000094,
"P79y": 238.89320075909347,
"P54y": 252.24900000000142,
"P54x": 431.5305000000024,
"P73x": 427.05,
"P73y": 191,
"P68y": 267.6976650000015,
"P30y": 214.61539500000094,
"P30x": 440.86117500000194,
"P14y": 243.47656317600092,
"P14x": 384.18704449200146,
"P63y": 254.87442000000144,
"P76occluded": true,
"P22x": 406.8646650000017,
"P22y": 176.94090000000077,
"P28occluded": true,
"P6y": 296.24299366950106,
"P6x": 367.5863697300013,
"P92x": 428.85,
"P38y": 193.3815000000009,
"P38x": 388.5255000000018,
"P94y": 188.5,
"P72y": 197.70000000000002,
"P72x": 395.65000000000003,
"P78y": 210.5218971000002,
"P63x": 427.8391200000024,
"P35occluded": true,
"P82x": 393.8000000000004,
"P82y": 200.95000000000022,
"P11occluded": true,
"tool-version": "1.0",
"P41y": 200.99550000000093,
"P41x": 396.5625000000018,
"P56occluded": true,
"P55x": 425.0508679558401,
"P55y": 259.9172483306748,
"P31x": 449.410005000002,
"P31y": 225.351135000001,
"P1y": 217.10946645000078,
"P75occluded": true,
"P62x": 420.38374500000236,
"P62y": 256.06728000000146,
"P15x": 373.5151821450014,
"P15y": 228.45690505800087,
"P49y": 261.4140000000014,
"P49x": 400.0875000000022,
"P25y": 170.87178263247637,
"P25x": 435.25400920037674,
"P2x": 311.0173699500011,
"P80occluded": true,
"P3y": 251.86940685000093,
"P39x": 397.33800000000184,
"P39y": 192.1830000000009,
"P69x": 394.6,
"P5x": 347.3103508088991,
"P5y": 287.4697160411496,
"P95x": 430,
"P95y": 189.25,
"P79x": 368.8999131564783,
"P57x": 434.7974700000025,
"P102x": 428.1,
"P102y": 190.85000000000002,
"P76x": 428.25
},
{
"l_eyex": 389.1221901922325,
"l_eyey": 197.94528259092206,
"tool-version": "1.0",
"l_status": "open",
"r_status": "occluded",
"r_eyex": 633.489814294182,
"r_eyey": 10.52527209626886,
"class": "eyes"
}
]
},
{
"filename": "data/001_03_01_130_05.png",
"class": "image",
"annotations": [
{
"face_outer_bboxy": 36.21548211860577,
"face_outer_bboxx": 259.54428851667467,
"face_tight_bboxx": 265.58020220310897,
"face_tight_bboxy": 116.19133846386018,
"tool-version": "1.0",
"face_tight_bboxwidth": 191.64025954428882,
"face_tight_bboxheight": 192.64624515869457,
"face_outer_bboxwidth": 198.68215884512887,
"Occlusionx": 0,
"class": "FaceBbox",
"face_outer_bboxheight": 273.62808711835464
},
{
"P91x": 283.35,
"P91y": 179.55,
"P28x": 304.14947850000084,
"P28y": 176.3226009000005,
"P5occluded": true,
"P52y": 244.28250000000094,
"P52x": 305.0535000000012,
"P32y": 220.38088500000066,
"P32x": 289.76557500000087,
"P44x": 334.8750000000012,
"P44y": 168.63600000000062,
"P99x": 340.20000000000005,
"P99y": 174.75,
"P75x": 343.90000000000003,
"P75y": 171.70000000000002,
"P20x": 269.9839800000006,
"P20y": 158.94859500000035,
"P8y": 299.437842994699,
"P8x": 301.7845345542186,
"P94x": 342.70000000000005,
"P12y": 272.68555921617576,
"P12x": 389.08146056834715,
"P65y": 249.500000000001,
"P65x": 321.9500000000013,
"P84x": 285.8,
"P84y": 175.5,
"P43y": 176.03850000000065,
"P43x": 329.9400000000012,
"P68x": 302.05,
"P68y": 252.55,
"P92y": 165.70000000000002,
"P92x": 343.40000000000003,
"P53x": 311.11650000000117,
"P53y": 241.18050000000093,
"P33x": 295.53106500000086,
"P33y": 224.95351500000066,
"P60x": 297.5100000000011,
"P60y": 258.382500000001,
"P23y": 149.55274915302633,
"P23x": 325.5457633496816,
"P90y": 177.15,
"P13x": 406.681647264744,
"P13y": 256.7280566114426,
"P7x": 292.6324374720922,
"P90x": 280.25,
"P58x": 309.7065000000012,
"P61y": 253.51800000000097,
"P104x": 346.0500000000002,
"P104y": 171.35000000000008,
"P83y": 174.70000000000002,
"P83x": 282.15000000000003,
"P61x": 296.31150000000116,
"P50y": 249.21750000000097,
"P50x": 294.19650000000115,
"P100x": 339.45000000000005,
"P100y": 171.75,
"P34y": 224.85411000000067,
"P34x": 300.8989350000009,
"P18y": 170.97660000000036,
"P18x": 268.59231000000057,
"P46x": 357.7170000000013,
"P46y": 172.86600000000064,
"P58y": 258.664500000001,
"P4occluded": true,
"P77x": 300.22496910000007,
"P77y": 221.17413690000006,
"tool-version": "1.0",
"P10y": 298.73383552684317,
"P10x": 341.67829106605154,
"P26x": 361.5470170921228,
"P26y": 148.3723801778643,
"version": "v1",
"P86x": 286.6,
"P86y": 181.9,
"P2y": 204.16216567820388,
"P2x": 300.6111887744588,
"P29y": 189.63790065000055,
"P29x": 301.90690170000084,
"P101y": 168.8,
"P101x": 340.40000000000003,
"P51x": 298.49700000000115,
"P51y": 243.57750000000092,
"P35x": 310.83943500000095,
"P35y": 223.36303500000068,
"P66x": 313.85,
"P66y": 251.05,
"P19x": 267.8964750000006,
"P19y": 165.11170500000037,
"P98y": 176.55,
"P98x": 342.85,
"P45y": 165.8865000000006,
"P45x": 347.2125000000013,
"P21y": 156.26466000000033,
"P21x": 276.4453050000006,
"P59x": 303.2910000000012,
"P59y": 258.664500000001,
"P9x": 316.33402222324,
"P9y": 303.89655695778623,
"P17occluded": true,
"P11x": 366.0838832850552,
"P11y": 286.0617011054374,
"P70y": 178.35000000000002,
"P70x": 283.15000000000003,
"P1x": 307.6512634530176,
"P1y": 189.14333969727852,
"P85y": 178.60000000000002,
"P85x": 287.5,
"P69y": 179.3,
"P69x": 282.65000000000003,
"P36y": 219.78445500000066,
"P36x": 326.446020000001,
"P77occluded": true,
"P81y": 173.25,
"P81x": 281.95,
"P40x": 298.00350000000105,
"P40y": 178.85850000000062,
"P71x": 283.95,
"P71y": 178.9,
"P56y": 254.505000000001,
"P56x": 323.24250000000126,
"P7y": 284.1843478578217,
"P89y": 179.85000000000002,
"P89x": 279.90000000000003,
"P48x": 338.96400000000125,
"P48y": 177.51900000000066,
"P16y": 205.1008423020117,
"P16x": 420.9964657778135,
"P24x": 338.1757113839151,
"P24y": 146.9559374076699,
"class": "FiducialPoints",
"P74y": 170.85000000000002,
"P4y": 234.6691559519585,
"P4x": 290.9897533804285,
"P96y": 172,
"P74x": 342.95000000000005,
"P3occluded": true,
"P78occluded": true,
"P103y": 179.4,
"P103x": 285.55,
"P80x": 444.78450000000055,
"P80y": 173.78250000000023,
"P37x": 275.44350000000094,
"P37y": 182.80650000000063,
"P47y": 176.95500000000064,
"P47x": 350.5965000000013,
"P64x": 313.45000000000124,
"P64y": 249.95000000000098,
"P76y": 172.65,
"P57y": 257.113500000001,
"P6occluded": true,
"P88x": 281.1,
"P88y": 182.60000000000002,
"P17x": 420.2924583099576,
"P17y": 187.96999391751874,
"P96x": 347.70000000000005,
"P67y": 252.45000000000002,
"P27y": 153.68404056609333,
"P27x": 370.04567371328926,
"P87y": 183.5,
"P87x": 283.95,
"P3x": 295.44846734351574,
"P67x": 307.95000000000005,
"P2occluded": true,
"P97x": 346.25,
"P97y": 175.35000000000002,
"P93x": 344.35,
"P93y": 177.8,
"P42x": 280.30800000000096,
"P42y": 185.34450000000064,
"P54y": 243.78900000000093,
"P54x": 321.0570000000012,
"P73x": 342.5,
"P73y": 171.95000000000002,
"P30y": 201.83191200000059,
"P30x": 298.26271440000085,
"P14y": 239.83187738290155,
"P14x": 416.77242097067824,
"P63y": 251.20000000000098,
"P63x": 307.7500000000012,
"P22x": 284.2983000000006,
"P22y": 157.95454500000034,
"P1occluded": true,
"P6y": 270.10419850070423,
"P6x": 289.34706928876477,
"P38y": 176.17950000000062,
"P38x": 277.06500000000096,
"P94y": 166.65,
"P72y": 180.35000000000002,
"P72x": 283.65000000000003,
"P78y": 176.32260090000005,
"P78x": 318.5860666500001,
"P82x": 285.6,
"P82y": 185.35000000000002,
"P32occluded": true,
"P41y": 183.37050000000062,
"P41x": 290.460000000001,
"P55x": 334.9455000000013,
"P55y": 251.89650000000097,
"P31x": 295.59965445000086,
"P31y": 213.04479600000062,
"P79y": 232.39037411700184,
"P62x": 301.6000000000012,
"P62y": 251.30000000000098,
"P15x": 420.05778915400566,
"P15y": 222.93569815436055,
"P49y": 256.690500000001,
"P49x": 291.65850000000114,
"P25y": 145.8936053300241,
"P25x": 350.45154872559993,
"P3y": 218.94632250317727,
"P39x": 286.30050000000097,
"P39y": 173.50050000000059,
"P5x": 288.8777309768609,
"P5y": 253.44268842811516,
"P95x": 346.6,
"P95y": 168.25,
"P79x": 431.0439612134159,
"P57x": 315.4875000000012,
"P102x": 343.1,
"P102y": 172,
"P76x": 343.20000000000005
},
{
"l_eyex": 289.90000000000003,
"l_eyey": 179.60000000000002,
"tool-version": "1.0",
"l_status": "open",
"r_status": "open",
"r_eyex": 337.4000000000001,
"r_eyey": 173.35000000000005,
"class": "eyes"
}
]
}