- Email: [email protected]
Solution for Sound Playback Delay on Android Devices
Available online at www.sciencedirect.com
ScienceDirect Procedia Computer Science 104 (2017) 413 – 420
ICTE 2016, December 2016, Riga, Latvia
Solution for Sound Playback Delay on Android Devices Dmitrijs Zigunovsa, Jekaterina Smirnovaa,*, Gatis Vitolsa, Gintautas Stonysb a
Latvia University of Agriculture, 2 Liela Street, Jelgava, LV-3001, Latvia b Alytaus kolegija, 17 Studentu Street, Alytus, 62252, Lithuania
Abstract Recent reports show that there is a 40-200 millisecond delay for sound playback in Android devices. At the same time there is no such reports and identified problem for other platforms. Research focuses on unresolved audio playback latency problems in devices with Android operating systems. Examination of possible problems, application of audio playback latency recording and measuring methods and audio playback latency decreasing methods are used to achieve a smallest possible audio playback delay. As a result with a proposed solution audio playback delay was reduced to 9ms using OpenSL EU API application and uncoded audio data with a device that support digitization and most high performance. © 2017 by by Elsevier B.V.B.V. This is an open access article under the CC BY-NC-ND license © 2016The TheAuthors. Authors.Published Published Elsevier (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016. Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016
Keywords: Audio playback latency; Android devices; Mobile application development
1. Introduction Audio playback impact all layers of application processing, initially with giving the command in the software application framework to the audio playback, for example in Linux kernel. Use of Android Software Development Kit (SDK) is appropriate way to implement audio file playback to devices with the Android Operation System (OS)1. Literature2,3 confirm that there is on average between 40-200 milisecond (ms) sound delay on Android devices. Sometimes delay exceeds 200ms. However such problem has not been identified in devices with other OS4. It could be that this issue is Android OS related. Usually a person feels sound delay around 10-15 ms and above5. Therefore any developed product sound playback delay should not exceed this level.
* Corresponding author. Tel.: +371 63005701; fax: +371 63023095. E-mail address: [email protected]
1877-0509 © 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016 doi:10.1016/j.procs.2017.01.154
414
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
Android OS has the following playback methods6: x Android.media.MediaPlayer object through the Android SDK (from Application framework layer) x EU OpenSL API via Android Native Development Kit (NDK) (Android service layer) To playback a simple audio file, for example when developing audio players, MediaPlayer object typically is used. This object provides easy control to playback process using recommended parameters, such as the size of the buffer and the decoding algorithm. Programmers usually need to configure playback process in a local Android environment using Application Programming Interface (API) such as Open Sound Library for Embedded Systems (OpenSL ES). This gives more control to playback process, but also increases development complexity. Use of this API is based on multimedia application development requiring the smallest possible sound delay time, or necessity to create cross-platform code. The aim of this research is to ensure a smallest possible audio playback delay in devices with Android operating system. The experimental part of the research includes testing audio playback during application development with various audio data types, audio playback delay measurement and data analysis. 2. Materials and methods There are several measurement options with different levels of measuring accuracy and application complexity. But all of these measurement methods are based on the idea that it is necessary to register time of the command giving moment and the time of the moment when the sound starts to play. Moreover, these time moments must be registered in one time metering system. Therefore, commands giving process must be conveyed as observation result. For this research light-emitting diode can be used. The smallest delay (less than 0.5 milliseconds) between giving command and monitoring can be recorded with the switch of the LED indicator. If simultaneously commands of switching on LED and audio file playback through dynamics or TRS connector are given, delay between LED indicator and sound playback can be recorded. Signal must be passed to dual-channel oscilloscope that records LED indicator state and the playback state. LED indicator activation can be converted into a signal using a photoelectric cell, which must be placed on opposite side to the LED indicator. The photoelectric signal channel must be connected to one of the oscilloscope channels. LED start-up time takes about 2 ms. This time depends on the LED specification7. Also additional photoelectric response time must be taken into consideration which usually is between 5 ms and 100 ms. This time have correlation with light brightness. When light is brighter, photoelectric response time is faster. As modern LED devices are able to produce up to 100 lux and more, the photoelectric response time will be minimal. However, for this experimental setting hardware combination adds totally to about 7 ms to playback command signal fixing time. It should be taken into consideration when processing the recordings. If there is an access to the Printed Circuit Board (PCB) it is possible to use it to connect LED to the oscilloscope channel. This will remove the created delay and will increase the accuracy of the result.
(a)
(b)
Fig. 1. (a) Typical scheme for light switch observation; (b) Actual scheme for light switch observation.
415
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
Photoelectric cell voltage is observed using voltmeter (see Fig. 1(a).) where R1 – resistor, Rf – a photoelectric cell, RV – voltmeter resistance, U– voltage source, V– voltmeter, GND – grounding. As circuit elements are connected in series, R1 and U are constants, then the voltage of the photoelectric cell varies with the resistance, which in turn varies with the incident light intensity. This voltage (Vf) is calculated using the formula (1).
Vf
U Rf R f R1
.
(1)
In reality, the voltage measuring devices have their constant resistance. Therefore, the real light switching on observation scheme is showed in Fig. 1(b). With the known voltmeter resistance voltage (Vf) is calculated using the formula (2).
Vf
U
U R1 Rv R f R1 Rv R f
(2)
Knowing the photoelectric cell resistance dependence on light intensity and tested device LED light intensity, voltage to the photoelectric cell can be calculated when the LED is turned on and off. Using these values, the indicator, which indicates when LED is turned on, can be identified. An indicator can take a situation where the voltage of the photoelectric cell falls in range from ʹΤ͵ ή ሺ୫ୟ୶ െ ܸ ሻ and ͳΤ͵ ή ሺܸ௫ െ ܸ ሻ having background voltage into consideration. The voltage in circuit should not be too low. 3. Experimental setting and testing application To get the experimental results it was necessary to connect oscilloscope channel to the sound TRS audio output channel. The developed scheme was used for final testing (see Fig. 2).
Fig. 2. Common playback delay observation scheme.
To get objective results of audio playback delay, measurements were performed on multiple devices: Sony Xperia V, LG G3, Just5 Blaster 2. These devices vary with Android OS version, branding, and technical support and cover majority of Android OS modifications at the moment of the experiment. An oscilloscope with a time digitization and functionality of measuring time delay in milliseconds is needed to delay time measuring in testing smartphones. Therefore, selected oscilloscope must be able to perform measurements with a frequency of 1 kHz or faster. The requirement is also two-channel availability and the ability to record the direct current. SEINTEK S2405 digital oscilloscope was used for delay time measuring in testing smartphones.
416
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
For testing purpose Android application was developed. Following audio file playback methods were implemented into the developed application: x Android.media.MediaPlayer object via Android SDK x OpenSL ES via Android NDK x Superpowered Audio API via Android NDK Immediately after application launcher the necessary objects are initialized, configured and prepared for work. The application provides the control using graphical interface (see Fig. 3). Each method complies with a button, which provides playback start command. In this way there is no activity related to the playback start command giving and finally no linked playback delay.
Fig. 3. Developed application interface for research purposes.
Immediately after the sound playback, application calls the LED command. Both these commands are given in a separate threads. 4. Choosing formats for the sound testing MP3 is the most popular and more distributed file format for sounds storage. Audio data in this format is encoded with losses. The data must be encoded at first to playback MP3 type audio files. In this situation playback delay occurs due to decoding process. To observe the audio data storage format impact on the playback delay time, it was necessary to make tests using open-label uncompressed audio data formats. The way to play such data with all the methods is use of Waveform Audio File Format (WAV) container with Pulse-code modulation (PCM) format data. To know exactly the audio data storage format impact on the play amount of delay, the third format was used, which is supported by all the used testing methods. As the third format Advanced Audio Coding (AAC) format was chosen because it is the modernized version of the standard MP3 format. This audio data encoding compresses them over the same MP3 sound quality which in result produces ACC format sound with better quality than MP3 with the same size of data file.
417
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
5. Results and discussion The results (see Table 1) were grouped by playbacks methods, devices, and audio data. Table 1. Playback delay measurements (milliseconds). LGG3 Playback method
Data format
44.1kHzfrequency
MediaPlayer
AAC
50,520
OpenSLES
Superpowered
37,406
48kHzfrequency 47,032
52,744
60,482
23,222
48,338
49,275
49,210
57,664
MP3
89,476
43,086
42,180
47,351
45,174
37,484
44,824
47,400
99,758
24,741
PCM
37,940
44,693
49,478
54,898
50,262
19,870
22,850
12,407
54,939
54,334
AAC
114,697
72,160
69,408
105,153
74,201
87,186
49,856
57,843
47,267
72,054
MP3
104,881
47,596
60,379
49,935
50,224
51,831
47,904
48,222
49,137
47,532
PCM
55,331
49,435
17,592
42,233
39,931
11,977
12,517
9,722
62,674
60,141
AAC
91,660
97,369
107,431
107,961
107,408
102,802
100,761
94,804
61,731
94,358
MP3
99,411
98,255
92,090
99,723
98,084
47,068
45,041
99,317
34,989
72,280
PCM
36,832
45,170
87,571
62,896
62,765
44,775
47,544
47,501
50,050
48,335
200,207
197,478
203,003
Just5Blaster2 Playback method
Data format
44.1kHzfrequency
MediaPlayer
AAC
179,262
OpenSLES
Superpowered
180,890
48kHzfrequency 180,657
175,183
185,006
194,880
187,522
MP3
145,209
162,081
154,208
152,509
154,991
174,368
170,122
171,817
177,812
180,330
PCM
122,716
137,532
149,284
134,625
127,305
135,235
140,048
155,536
150,212
152,562
AAC
149,647
180,472
177,886
154,768
167,945
174,802
162,083
167,842
174,833
177,727
MP3
150,036
155,080
152,041
162,496
164,885
160,130
162,858
171,875
169,142
169,648
PCM
134,924
138,285
129,221
141,744
145,129
149,856
152,923
148,242
150,070
149,583
AAC
424,409
436,802
427,528
430,195
421,945
435,638
432,307
449,796
429,680
434,906
MP3
396,587
402,905
385,593
407,501
422,529
412,185
400,079
385,011
402,853
424,950
PCM
327,069
323,266
306,958
342,307
347,202
330,404
353,262
345,459
351,866
355,504
SonyXperiaV Playback method MediaPlayer
OpenSLES
Superpowered
Data format
44.1kHzfrequency
48kHzfrequency
AAC
344,937
345,528
349,963
239,148
350,154
194,384
344,442
350,226
210,266
374,209
MP3
220,183
294,427
285,063
299,701
294,648
190,531
290,439
299,856
199,985
284,579
PCM
179,978
300,593
300,179
289,395
319,614
199,702
274,986
279,500
274,813
290,062
AAC
254,576
220,343
225,385
285,587
239,145
244,978
194,906
219,565
234,992
205,443
MP3
295,111
290,134
190,373
240,180
184,318
229,485
194,965
185,331
194,552
184,499
PCM
224,890
169,821
180,473
189,682
179,990
245,073
169,782
185,558
170,019
165,815
AAC
499,633
500,038
489,949
510,057
519,792
480,806
489,226
490,046
520,111
479,388
MP3
510,070
480,146
479,481
499,809
489,322
470,017
499,686
499,905
489,546
480,226
PCM
430,132
440,744
440,734
420,485
430,132
440,248
439,735
419,542
409,376
440,005
418
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
Because all measurements were made directly, measurement error was calculated using formula (3) where Ɂ – relative mean error, %; – measuring device absolute error, V; ୶ – the mean value of the root mean square error (4), V; – measurements count; Ƚ – importance rate, Ƚ אሾͲǢ ͳሿ; ǡ୬ିଵ – student's t-distribution; ୟ୴ – arithmetic average of the n measurements, V. ) 12,13.
ߜൌ
ටሺௌೣ ௧ഀǡషభ ሻమ ାమ
ܵ௫ ൌ ට
(3)
௫ೌೡ మ σ సభሺ௫ೌೡ ି௫ ሻ
ሺିଵሻ
(4)
Generally relative measurement error is accepted as broad, because each research which measure different data, allow special maximum relative error rate. This rate permits to determine the reliability of measurement results. Performed playback delay measurement can raise a subjective agreement that the results with a relative error of more than 50% are not reliable. The maximum relative error is accepted with the agreement, that the used method of measurement is accurate. But background processes which may be affected to the result of the obtained values are included in solution with tested devices.
Fig. 4. Relative errors measurement.
As a result, measurement errors are different (see Fig. 4). There is no assurance that the results reflect the real situation. One factor could be that device loading is impacted by the background running software. In addition, the device load increases with use of Wi-Fi, Global System for Mobile Communications (GSM), Near-field communication (NFC), Global Positioning System (GPS), Bluetooth. The greater is the device loading level, the more load it takes between sound processing cycles. Since the test device LG G3 was under increased load measurements, it could show a greater distinction than measurements of other devices. In addition, testing using Just5 Blaster 2 device gave lowest relative error measurement because it was used only by default included software.
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
Playback delay average values by used data encoding type are different in test devices (see Fig. 5). Device specification impact to the audio playback delay is undeniable. When comparing average playback delay value used by the data encoding, it can be concluded that the PCM-format data is played back the fastest. This speed has been triggered with no need to decode the data. MP3 playback data decoding due delay to increases for 27.64 ms comparing with PCM audio data playback testing. Tested AAC data decoding takes about 17.56 ms more time than the MP3 data decoding because AAC encoding algorithm execution takes more time.
Fig. 5. Data format impact to audio playback delay.
Fig. 6. Supported and unsupported audio digitization playback delay difference.
Sounds discretization also impacts performance time and computing resource. This factor also increases playback delay time. This conclusion is illustrated in the test results (see Fig. 6).
419
420
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
It can be observed that test devices playback delay difference is small for data with different discretization. 6. Conclusion Audio playback delay was reduced using OpenSL EU API application, uncoded audio data in the device supported digitization. Using developed application it is possible to reduce the sound delay time to 9ms that does not interfere to human perception. Uncoded data playback is faster than the coded data playback. Data is processed faster using digitization that is supported by devices rather than using other digitization method. High performance devices have smallest playback delay time. Testing process can be improved by reducing measurement errors using factory setting Android operation system which is free from additionally installed and running software, turned off communication modules and full battery connected to the charging network. Such conditions are close to the ideal test conditions, however real living situation such conditions are not present. References 1. 2. 3. 4. 5. 6. 1. 2. 3.
4. 5. 6. 7.
Android Open Source Project. Audio. Available: https://source.android.com/devices/audio/. BlackBerry Developer Blog. Tone Generatorௗ: How to add Low Latency Audio to your Native or Cascades App. July 2013. Available: http://devblog.blackberry.com/2013/07/tone-generator/. Superpowered.com. Android Audio Latency and iOS Audio Latency Test App. Available: http://superpowered.com/latency/#table. Lobao M. Android Audio Latency In-Depth: It’s Getting Better, Especially With The Nexus 5X And 6P. International Telecommunication Union. Recommendation BT.1359: Relative Timing of Sound and Vision for Broadcasting. Geneva, Switzerland; 1998. Huang CC, Nie R. Design and realization of music player based on android system. In: Int Conf Mechatronics Inf Technol. Vol. 462; 2013. p. 1123-1126. Khan LU. Visible light communication: Applications, architecture, standardization and research challenges. Digit Commun Networks; 2016. Zheng H, Chen J, Yu C, Gurusamy M. Inverse design of LED arrangement for visible light communication systems. Opt Commun.; 2017. Bolton JR. University of Western Ontario. Photochemistry Unit., Boston University. Department of Chemistry. Solar Power and Fuelsௗ: In: Proceedings of the First International Conference on the Photochemical Conversion and Storage of Solar Energy. London, Canada. Academic Press; 1977. Henderson R. Samsung’s new LED components could improve Galaxy S5 flash and hint at April launch; 2014. Available: http://www.pocket-lint.com/news/127328-samsung-s-new-led-components-could-improve-galaxy-s5-flash-and-hint-at-april-launch. Rumsey F, McCormick T. Sound and Recording. Elsevier/Focal; 2009. Myttenaere A, Golden B, Le Grand B, Rossi F. Mean Absolute Percentage Error for regression models. Neurocomputing. 192; 2016. p. 38-48. Nakajima J, Omori Y. Stochastic volatility model with leverage and asymmetrically heavy-tailed error using GH skew Student’s tdistribution. Comput Stat Data Anal. 56(11); 2012. p. 3690-3704.
Gatis Vitols received PhD in Information technologies at Latvia University of Agriculture. Primary research fields include cross-cultural information systems development, usability and database technologies. Author works as a leading researcher and associate professor at Faculty of Information Technologies in Latvia University of Agriculture. Contact him at [email protected].
ScienceDirect Procedia Computer Science 104 (2017) 413 – 420
ICTE 2016, December 2016, Riga, Latvia
Solution for Sound Playback Delay on Android Devices Dmitrijs Zigunovsa, Jekaterina Smirnovaa,*, Gatis Vitolsa, Gintautas Stonysb a
Latvia University of Agriculture, 2 Liela Street, Jelgava, LV-3001, Latvia b Alytaus kolegija, 17 Studentu Street, Alytus, 62252, Lithuania
Abstract Recent reports show that there is a 40-200 millisecond delay for sound playback in Android devices. At the same time there is no such reports and identified problem for other platforms. Research focuses on unresolved audio playback latency problems in devices with Android operating systems. Examination of possible problems, application of audio playback latency recording and measuring methods and audio playback latency decreasing methods are used to achieve a smallest possible audio playback delay. As a result with a proposed solution audio playback delay was reduced to 9ms using OpenSL EU API application and uncoded audio data with a device that support digitization and most high performance. © 2017 by by Elsevier B.V.B.V. This is an open access article under the CC BY-NC-ND license © 2016The TheAuthors. Authors.Published Published Elsevier (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016. Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016
Keywords: Audio playback latency; Android devices; Mobile application development
1. Introduction Audio playback impact all layers of application processing, initially with giving the command in the software application framework to the audio playback, for example in Linux kernel. Use of Android Software Development Kit (SDK) is appropriate way to implement audio file playback to devices with the Android Operation System (OS)1. Literature2,3 confirm that there is on average between 40-200 milisecond (ms) sound delay on Android devices. Sometimes delay exceeds 200ms. However such problem has not been identified in devices with other OS4. It could be that this issue is Android OS related. Usually a person feels sound delay around 10-15 ms and above5. Therefore any developed product sound playback delay should not exceed this level.
* Corresponding author. Tel.: +371 63005701; fax: +371 63023095. E-mail address: [email protected]
1877-0509 © 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016 doi:10.1016/j.procs.2017.01.154
414
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
Android OS has the following playback methods6: x Android.media.MediaPlayer object through the Android SDK (from Application framework layer) x EU OpenSL API via Android Native Development Kit (NDK) (Android service layer) To playback a simple audio file, for example when developing audio players, MediaPlayer object typically is used. This object provides easy control to playback process using recommended parameters, such as the size of the buffer and the decoding algorithm. Programmers usually need to configure playback process in a local Android environment using Application Programming Interface (API) such as Open Sound Library for Embedded Systems (OpenSL ES). This gives more control to playback process, but also increases development complexity. Use of this API is based on multimedia application development requiring the smallest possible sound delay time, or necessity to create cross-platform code. The aim of this research is to ensure a smallest possible audio playback delay in devices with Android operating system. The experimental part of the research includes testing audio playback during application development with various audio data types, audio playback delay measurement and data analysis. 2. Materials and methods There are several measurement options with different levels of measuring accuracy and application complexity. But all of these measurement methods are based on the idea that it is necessary to register time of the command giving moment and the time of the moment when the sound starts to play. Moreover, these time moments must be registered in one time metering system. Therefore, commands giving process must be conveyed as observation result. For this research light-emitting diode can be used. The smallest delay (less than 0.5 milliseconds) between giving command and monitoring can be recorded with the switch of the LED indicator. If simultaneously commands of switching on LED and audio file playback through dynamics or TRS connector are given, delay between LED indicator and sound playback can be recorded. Signal must be passed to dual-channel oscilloscope that records LED indicator state and the playback state. LED indicator activation can be converted into a signal using a photoelectric cell, which must be placed on opposite side to the LED indicator. The photoelectric signal channel must be connected to one of the oscilloscope channels. LED start-up time takes about 2 ms. This time depends on the LED specification7. Also additional photoelectric response time must be taken into consideration which usually is between 5 ms and 100 ms. This time have correlation with light brightness. When light is brighter, photoelectric response time is faster. As modern LED devices are able to produce up to 100 lux and more, the photoelectric response time will be minimal. However, for this experimental setting hardware combination adds totally to about 7 ms to playback command signal fixing time. It should be taken into consideration when processing the recordings. If there is an access to the Printed Circuit Board (PCB) it is possible to use it to connect LED to the oscilloscope channel. This will remove the created delay and will increase the accuracy of the result.
(a)
(b)
Fig. 1. (a) Typical scheme for light switch observation; (b) Actual scheme for light switch observation.
415
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
Photoelectric cell voltage is observed using voltmeter (see Fig. 1(a).) where R1 – resistor, Rf – a photoelectric cell, RV – voltmeter resistance, U– voltage source, V– voltmeter, GND – grounding. As circuit elements are connected in series, R1 and U are constants, then the voltage of the photoelectric cell varies with the resistance, which in turn varies with the incident light intensity. This voltage (Vf) is calculated using the formula (1).
Vf
U Rf R f R1
.
(1)
In reality, the voltage measuring devices have their constant resistance. Therefore, the real light switching on observation scheme is showed in Fig. 1(b). With the known voltmeter resistance voltage (Vf) is calculated using the formula (2).
Vf
U
U R1 Rv R f R1 Rv R f
(2)
Knowing the photoelectric cell resistance dependence on light intensity and tested device LED light intensity, voltage to the photoelectric cell can be calculated when the LED is turned on and off. Using these values, the indicator, which indicates when LED is turned on, can be identified. An indicator can take a situation where the voltage of the photoelectric cell falls in range from ʹΤ͵ ή ሺ୫ୟ୶ െ ܸ ሻ and ͳΤ͵ ή ሺܸ௫ െ ܸ ሻ having background voltage into consideration. The voltage in circuit should not be too low. 3. Experimental setting and testing application To get the experimental results it was necessary to connect oscilloscope channel to the sound TRS audio output channel. The developed scheme was used for final testing (see Fig. 2).
Fig. 2. Common playback delay observation scheme.
To get objective results of audio playback delay, measurements were performed on multiple devices: Sony Xperia V, LG G3, Just5 Blaster 2. These devices vary with Android OS version, branding, and technical support and cover majority of Android OS modifications at the moment of the experiment. An oscilloscope with a time digitization and functionality of measuring time delay in milliseconds is needed to delay time measuring in testing smartphones. Therefore, selected oscilloscope must be able to perform measurements with a frequency of 1 kHz or faster. The requirement is also two-channel availability and the ability to record the direct current. SEINTEK S2405 digital oscilloscope was used for delay time measuring in testing smartphones.
416
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
For testing purpose Android application was developed. Following audio file playback methods were implemented into the developed application: x Android.media.MediaPlayer object via Android SDK x OpenSL ES via Android NDK x Superpowered Audio API via Android NDK Immediately after application launcher the necessary objects are initialized, configured and prepared for work. The application provides the control using graphical interface (see Fig. 3). Each method complies with a button, which provides playback start command. In this way there is no activity related to the playback start command giving and finally no linked playback delay.
Fig. 3. Developed application interface for research purposes.
Immediately after the sound playback, application calls the LED command. Both these commands are given in a separate threads. 4. Choosing formats for the sound testing MP3 is the most popular and more distributed file format for sounds storage. Audio data in this format is encoded with losses. The data must be encoded at first to playback MP3 type audio files. In this situation playback delay occurs due to decoding process. To observe the audio data storage format impact on the playback delay time, it was necessary to make tests using open-label uncompressed audio data formats. The way to play such data with all the methods is use of Waveform Audio File Format (WAV) container with Pulse-code modulation (PCM) format data. To know exactly the audio data storage format impact on the play amount of delay, the third format was used, which is supported by all the used testing methods. As the third format Advanced Audio Coding (AAC) format was chosen because it is the modernized version of the standard MP3 format. This audio data encoding compresses them over the same MP3 sound quality which in result produces ACC format sound with better quality than MP3 with the same size of data file.
417
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
5. Results and discussion The results (see Table 1) were grouped by playbacks methods, devices, and audio data. Table 1. Playback delay measurements (milliseconds). LGG3 Playback method
Data format
44.1kHzfrequency
MediaPlayer
AAC
50,520
OpenSLES
Superpowered
37,406
48kHzfrequency 47,032
52,744
60,482
23,222
48,338
49,275
49,210
57,664
MP3
89,476
43,086
42,180
47,351
45,174
37,484
44,824
47,400
99,758
24,741
PCM
37,940
44,693
49,478
54,898
50,262
19,870
22,850
12,407
54,939
54,334
AAC
114,697
72,160
69,408
105,153
74,201
87,186
49,856
57,843
47,267
72,054
MP3
104,881
47,596
60,379
49,935
50,224
51,831
47,904
48,222
49,137
47,532
PCM
55,331
49,435
17,592
42,233
39,931
11,977
12,517
9,722
62,674
60,141
AAC
91,660
97,369
107,431
107,961
107,408
102,802
100,761
94,804
61,731
94,358
MP3
99,411
98,255
92,090
99,723
98,084
47,068
45,041
99,317
34,989
72,280
PCM
36,832
45,170
87,571
62,896
62,765
44,775
47,544
47,501
50,050
48,335
200,207
197,478
203,003
Just5Blaster2 Playback method
Data format
44.1kHzfrequency
MediaPlayer
AAC
179,262
OpenSLES
Superpowered
180,890
48kHzfrequency 180,657
175,183
185,006
194,880
187,522
MP3
145,209
162,081
154,208
152,509
154,991
174,368
170,122
171,817
177,812
180,330
PCM
122,716
137,532
149,284
134,625
127,305
135,235
140,048
155,536
150,212
152,562
AAC
149,647
180,472
177,886
154,768
167,945
174,802
162,083
167,842
174,833
177,727
MP3
150,036
155,080
152,041
162,496
164,885
160,130
162,858
171,875
169,142
169,648
PCM
134,924
138,285
129,221
141,744
145,129
149,856
152,923
148,242
150,070
149,583
AAC
424,409
436,802
427,528
430,195
421,945
435,638
432,307
449,796
429,680
434,906
MP3
396,587
402,905
385,593
407,501
422,529
412,185
400,079
385,011
402,853
424,950
PCM
327,069
323,266
306,958
342,307
347,202
330,404
353,262
345,459
351,866
355,504
SonyXperiaV Playback method MediaPlayer
OpenSLES
Superpowered
Data format
44.1kHzfrequency
48kHzfrequency
AAC
344,937
345,528
349,963
239,148
350,154
194,384
344,442
350,226
210,266
374,209
MP3
220,183
294,427
285,063
299,701
294,648
190,531
290,439
299,856
199,985
284,579
PCM
179,978
300,593
300,179
289,395
319,614
199,702
274,986
279,500
274,813
290,062
AAC
254,576
220,343
225,385
285,587
239,145
244,978
194,906
219,565
234,992
205,443
MP3
295,111
290,134
190,373
240,180
184,318
229,485
194,965
185,331
194,552
184,499
PCM
224,890
169,821
180,473
189,682
179,990
245,073
169,782
185,558
170,019
165,815
AAC
499,633
500,038
489,949
510,057
519,792
480,806
489,226
490,046
520,111
479,388
MP3
510,070
480,146
479,481
499,809
489,322
470,017
499,686
499,905
489,546
480,226
PCM
430,132
440,744
440,734
420,485
430,132
440,248
439,735
419,542
409,376
440,005
418
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
Because all measurements were made directly, measurement error was calculated using formula (3) where Ɂ – relative mean error, %; – measuring device absolute error, V; ୶ – the mean value of the root mean square error (4), V; – measurements count; Ƚ – importance rate, Ƚ אሾͲǢ ͳሿ; ǡ୬ିଵ – student's t-distribution; ୟ୴ – arithmetic average of the n measurements, V. ) 12,13.
ߜൌ
ටሺௌೣ ௧ഀǡషభ ሻమ ାమ
ܵ௫ ൌ ට
(3)
௫ೌೡ మ σ సభሺ௫ೌೡ ି௫ ሻ
ሺିଵሻ
(4)
Generally relative measurement error is accepted as broad, because each research which measure different data, allow special maximum relative error rate. This rate permits to determine the reliability of measurement results. Performed playback delay measurement can raise a subjective agreement that the results with a relative error of more than 50% are not reliable. The maximum relative error is accepted with the agreement, that the used method of measurement is accurate. But background processes which may be affected to the result of the obtained values are included in solution with tested devices.
Fig. 4. Relative errors measurement.
As a result, measurement errors are different (see Fig. 4). There is no assurance that the results reflect the real situation. One factor could be that device loading is impacted by the background running software. In addition, the device load increases with use of Wi-Fi, Global System for Mobile Communications (GSM), Near-field communication (NFC), Global Positioning System (GPS), Bluetooth. The greater is the device loading level, the more load it takes between sound processing cycles. Since the test device LG G3 was under increased load measurements, it could show a greater distinction than measurements of other devices. In addition, testing using Just5 Blaster 2 device gave lowest relative error measurement because it was used only by default included software.
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
Playback delay average values by used data encoding type are different in test devices (see Fig. 5). Device specification impact to the audio playback delay is undeniable. When comparing average playback delay value used by the data encoding, it can be concluded that the PCM-format data is played back the fastest. This speed has been triggered with no need to decode the data. MP3 playback data decoding due delay to increases for 27.64 ms comparing with PCM audio data playback testing. Tested AAC data decoding takes about 17.56 ms more time than the MP3 data decoding because AAC encoding algorithm execution takes more time.
Fig. 5. Data format impact to audio playback delay.
Fig. 6. Supported and unsupported audio digitization playback delay difference.
Sounds discretization also impacts performance time and computing resource. This factor also increases playback delay time. This conclusion is illustrated in the test results (see Fig. 6).
419
420
Dmitrijs Zigunovs et al. / Procedia Computer Science 104 (2017) 413 – 420
It can be observed that test devices playback delay difference is small for data with different discretization. 6. Conclusion Audio playback delay was reduced using OpenSL EU API application, uncoded audio data in the device supported digitization. Using developed application it is possible to reduce the sound delay time to 9ms that does not interfere to human perception. Uncoded data playback is faster than the coded data playback. Data is processed faster using digitization that is supported by devices rather than using other digitization method. High performance devices have smallest playback delay time. Testing process can be improved by reducing measurement errors using factory setting Android operation system which is free from additionally installed and running software, turned off communication modules and full battery connected to the charging network. Such conditions are close to the ideal test conditions, however real living situation such conditions are not present. References 1. 2. 3. 4. 5. 6. 1. 2. 3.
4. 5. 6. 7.
Android Open Source Project. Audio. Available: https://source.android.com/devices/audio/. BlackBerry Developer Blog. Tone Generatorௗ: How to add Low Latency Audio to your Native or Cascades App. July 2013. Available: http://devblog.blackberry.com/2013/07/tone-generator/. Superpowered.com. Android Audio Latency and iOS Audio Latency Test App. Available: http://superpowered.com/latency/#table. Lobao M. Android Audio Latency In-Depth: It’s Getting Better, Especially With The Nexus 5X And 6P. International Telecommunication Union. Recommendation BT.1359: Relative Timing of Sound and Vision for Broadcasting. Geneva, Switzerland; 1998. Huang CC, Nie R. Design and realization of music player based on android system. In: Int Conf Mechatronics Inf Technol. Vol. 462; 2013. p. 1123-1126. Khan LU. Visible light communication: Applications, architecture, standardization and research challenges. Digit Commun Networks; 2016. Zheng H, Chen J, Yu C, Gurusamy M. Inverse design of LED arrangement for visible light communication systems. Opt Commun.; 2017. Bolton JR. University of Western Ontario. Photochemistry Unit., Boston University. Department of Chemistry. Solar Power and Fuelsௗ: In: Proceedings of the First International Conference on the Photochemical Conversion and Storage of Solar Energy. London, Canada. Academic Press; 1977. Henderson R. Samsung’s new LED components could improve Galaxy S5 flash and hint at April launch; 2014. Available: http://www.pocket-lint.com/news/127328-samsung-s-new-led-components-could-improve-galaxy-s5-flash-and-hint-at-april-launch. Rumsey F, McCormick T. Sound and Recording. Elsevier/Focal; 2009. Myttenaere A, Golden B, Le Grand B, Rossi F. Mean Absolute Percentage Error for regression models. Neurocomputing. 192; 2016. p. 38-48. Nakajima J, Omori Y. Stochastic volatility model with leverage and asymmetrically heavy-tailed error using GH skew Student’s tdistribution. Comput Stat Data Anal. 56(11); 2012. p. 3690-3704.
Gatis Vitols received PhD in Information technologies at Latvia University of Agriculture. Primary research fields include cross-cultural information systems development, usability and database technologies. Author works as a leading researcher and associate professor at Faculty of Information Technologies in Latvia University of Agriculture. Contact him at [email protected].