Why is subtracting these two times (in 1927) giving a strange result?

6 190

1 473

If I run the following program, which parses two date strings referencing times 1 second apart and compares them:

public static void main(String[] args) throws ParseException {
    SimpleDateFormat sf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");  
    String str3 = "1927-12-31 23:54:07";  
    String str4 = "1927-12-31 23:54:08";  
    Date sDt3 = sf.parse(str3);  
    Date sDt4 = sf.parse(str4);  
    long ld3 = sDt3.getTime() /1000;  
    long ld4 = sDt4.getTime() /1000;
    System.out.println(ld4-ld3);
}

The output is:

353

Why is ld4-ld3 not 1 (as I would expect from the one-second difference in the times), but 353?

If I change the dates to times 1 second later:

String str3 = "1927-12-31 23:54:08";  
String str4 = "1927-12-31 23:54:09";  

Then ld4-ld3 will be 1.


Java version:

java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Dynamic Code Evolution Client VM (build 0.2-b02-internal, 19.0-b04-internal, mixed mode)

Timezone(`TimeZone.getDefault()`):

sun.util.calendar.ZoneInfo[id="Asia/Shanghai",
offset=28800000,dstSavings=0,
useDaylight=false,
transitions=19,
lastRule=null]

Locale(Locale.getDefault()): zh_CN

Freewind

Posted 2011-07-27T08:15:58.380

Reputation: 75 759

56The real answer is to always, always use seconds since an epoch for logging, like the Unix epoch, with 64 bit integer representation (signed, if you want to allow stamps before the epoch). Any real-world time system has some non-linear, non-monotonic behaviour like leap hours or daylight savings. – Phil H – 2012-07-12T08:34:06.390

139

originally posted as Oracle Bug ID 7070044 on Jul 23 `11

– Arno – 2012-08-04T10:55:55.817

29As @Costi Ciudatu , I really wonder from which bug report did the OP dig this one up. I'm also pretty sure except for being a puzzle this is not useful to 99.9(add lots of 9 digits) percent of users. He does get the medal for the reputation "game". – Menelaos Bakopoulos – 2013-10-11T10:32:39.470

16Lol. They fixed it for jdk6 in 2011. Then, two years after that, they discovered that it should get fixed in jdk7, too.... fixed as of 7u25, of course, I did not find any hint in the release note. Sometimes I wonder how many bugs Oracle fixes and tells nobody about it for PR reasons. – user1050755 – 2014-02-14T21:03:03.123

6

A great video about these kind of things: http://www.youtube.com/watch?v=-5wpm-gesOY

– Thorbjørn Ravn Andersen – 2014-10-14T10:39:15.423

2

And another from the same guy, @ThorbjørnRavnAndersen: https://www.youtube.com/watch?v=Uqjg8Kk1HXo (Leap seconds). (This one is from Tom Scott's own YouTube channel, not from Computerphile.)

– TRiG – 2015-07-03T16:23:47.353

1@PhilH The Nice thing is, there will still be leap seconds. So even that do not work. – 12431234123412341234123 – 2017-03-16T19:19:17.280

17This might be a locale problem. – Thorbjørn Ravn Andersen – 2011-07-27T08:22:34.813

1305Did you really stumble upon that exact situation in a real-life scenario or was this question only meant to be a puzzler -- just for the fun of it ? – Costi Ciudatu – 2011-07-27T08:42:29.710

229@Costi Ciudatu: FWIW, I could easily imagine this coming up as the result of reducing a larger bug -- i.e., "Why are these two dates a year apart not exactly a year apart?" – Brooks Moses – 2011-07-30T01:43:46.687

Answers

9 965

It's a time zone change on December 31st in Shanghai.

See this page for details of 1927 in Shanghai. Basically at midnight at the end of 1927, the clocks went back 5 minutes and 52 seconds. So "1927-12-31 23:54:08" actually happened twice, and it looks like Java is parsing it as the later possible instant for that local date/time - hence the difference.

Just another episode in the often weird and wonderful world of time zones.

EDIT: Stop press! History changes...

The original question would no longer demonstrate quite the same behaviour, if rebuilt with version 2013a of TZDB. In 2013a, the result would be 358 seconds, with a transition time of 23:54:03 instead of 23:54:08.

I only noticed this because I'm collecting questions like this in Noda Time, in the form of unit tests... The test has now been changed, but it just goes to show - not even historical data is safe.

EDIT: History has changed again...

In TZDB 2014f, the time of the change has moved to 1900-12-31, and it's now a mere 343 second change (so the time between t and t+1 is 344 seconds, if you see what I mean).

EDIT: To answer a question around a transition at 1900... it looks like the Java timezone implementation treats all time zones as simply being in their standard time for any instant before the start of 1900 UTC:

import java.util.TimeZone;

public class Test {
    public static void main(String[] args) throws Exception {
        long startOf1900Utc = -2208988800000L;
        for (String id : TimeZone.getAvailableIDs()) {
            TimeZone zone = TimeZone.getTimeZone(id);
            if (zone.getRawOffset() != zone.getOffset(startOf1900Utc - 1)) {
                System.out.println(id);
            }
        }
    }
}

The code above produces no output on my Windows machine. So any time zone which has any offset other than its standard one at the start of 1900 will count that as a transition. TZDB itself has some data going back earlier than that, and doesn't rely on any idea of a "fixed" standard time (which is what getRawOffset assumes to be a valid concept) so other libraries needn't introduce this artificial transition.

Jon Skeet

Posted 2011-07-27T08:15:58.380

Reputation: 1 070 456

267@EdS. It actually only took him 15 minutes, the reason it shows a 16 minute difference is due to a slight timezone discrepancy on July 27th 2011 caused by how awesome Jon Skeet is. – JessieArr – 2013-10-18T16:37:09.667

2I understand the reasoning behind it, and why it's implemented like this. I personally think the result should be 'undetermined' at best. The way it's implemented now is as if the first occurrences of 23:54:08-23:59:59 never happened. 1 and 353 would both be a correct answer. Technically 2 different timezone's are subtracted which doesn't feel like correct behavior. – It's me ... Alex – 2014-10-15T14:35:49.507

2@It'sme...Alex: No, there's only one time zone here - with two different UTC offsets. A better API would allow you to determine which instant you meant when parsing an ambiguous local time, of course :) – Jon Skeet – 2014-10-15T14:37:54.940

2@Jon Skeet: ok, two different UTC offsets :-) ... But you understand what I mean. The implementation in java (and most likely other libraries as well) is a bit poor – It's me ... Alex – 2014-10-15T14:40:21.050

7@It'sme...Alex: Well java.util.Calendar and java.util.Date are certainly poor in a number of ways. In Noda Time, when you convert a LocalDateTime to a ZonedDateTime, you have to give an indication of how you want to handle ambiguous or skipped instants. – Jon Skeet – 2014-10-15T15:04:08.880

5Is there a complete listing of all such time corrections that have been done in all of history? – Kevin H. Lin – 2014-10-16T01:58:23.353

5

@KevinH.Lin: Well there's the time zone data at IANA: http://iana.org/timezones

– Jon Skeet – 2014-10-16T05:50:34.870

1

@JonSkeet If this answer is correct, then it might be wise to add it to your answer too, that way it's super up to date

– Dan Temple – 2014-10-30T17:45:37.780

5@DanTemple: That answer isn't really correct - the time zone database changed the transition time slightly, but if you edit the original program appropriately, you'll still see a discontinuity. – Jon Skeet – 2014-10-30T17:49:12.040

10

I now see why Tom Scott says to just not do this. https://www.youtube.com/watch?v=-5wpm-gesOY

– Ben Leggiero – 2015-05-15T01:11:58.030

2

Tom Scott also has a video on leap seconds @BenC.R.Leggiero: https://www.youtube.com/watch?v=Uqjg8Kk1HXo. (This one is from Tom Scott's own YouTube channel, not from Computerphile.)

– TRiG – 2015-07-03T16:24:28.440

@Jon Skeet, I am not able to understand the difference. You have both times from the same time zone, if you add or subtract time to a time zone, both times must get affected. Then difference should be only one second. Can you please explain with numbers? – Gops AB – 2015-09-13T17:10:50.003

2@GopsAB: What do you mean by "both times from the same time zone"? Ignore Shanghai for the moment - do you live in a time zone that observes daylight saving time? If so, what time does it change? That's a good way of seeing local time change by a different amount from UTC...] – Jon Skeet – 2015-09-13T17:13:41.140

1@JonSkeet Yes, I meant time1 and time2 in the program are from the same time zone. Isn't? – Gops AB – 2015-09-14T06:32:59.403

5@GopsAB: Yes, but I don't think that means what you think it means. "Same time zone" doesn't mean "same UTC offset". For example, I'm in the time zone identified as "Europe/London". Once a year, our local time jumps forward by an hour. Once a year, our local time falls back by an hour. In both cases, two universal times which are one second apart end up having local times which are much more than a second apart. – Jon Skeet – 2015-09-14T07:15:23.053

3@GopsAB: It's not true for local times, because local times go forward or back quite often, usually due to DST. Again - do you live in a culture with daylight saving time? And what do you mean by "both times will get changed"? It's really not clear what's confusing you. – Jon Skeet – 2015-09-17T15:09:29.743

I am from India, No DST used in India. What I understood the implementation is something like an array structure, where reducing all the elements by a time will not change the arithmetic operations. To be more clear, a = [1,2,3,4,5], a[1]-a[2] is 1. reduce a by 1,a will be [0,1,2,3,4] , again a[1]-a[0] gives the same answer. I hope it clears my understanding. – Gops AB – 2015-09-17T15:24:17.463

@GopsAB: Forget implementations and arrays - it sounds like you need to understand the difference between UTC and "time in a particular time zone" first. Even though India doesn't observe DST, do you understand how it works in the real world? For example, in the UK, on the day that the local time goes forward by an hour, the local time will go 00:59:58, 00:59:59, 02:00:00, 02:00:01. – Jon Skeet – 2015-09-17T15:32:46.187

3So, clocks went back 5:52 in Shanghai. So 11:54:08 is actually 00:00:00. So differentiating 11:54:07 from 11:54:08 actually works like 00:00:00 - 11:54:07. Am i right? And Day light saving actually reducing the clock to increase the day and decreasing to increase the night duration.Thanks – Gops AB – 2015-09-17T15:48:29.190

7@GopsAB: I wouldn't say "11:54:08 is actually 00:00:00", but the second occurrence of 11:54:08 is when 00:00:00 would have been, if the UTC offset hadn't changed. This isn't actually an example of DST, btw. DST isn't about changing the duration of night time / day time - it's just an attempt to make "typical working hours" occur in more of the daylight hours. Anyway, I think that's really enough discussion here... – Jon Skeet – 2015-09-17T15:54:30.913

1Well apparently you CAN change the past, at least if you are developing the JVM :) – Ordiel – 2016-01-07T05:40:16.947

11I'm more interested in the times between Jul 27 '11 at 8:15 (the datetime the question was asked).. and the time of Jul 27 '11 at 8:31 (the datetime the question was answered).. was this really a 16 min difference in how long it took Jon Skeet to know this answer AND explain it? Shouldn't this be much longer? Or am I really underestimating Jon Skeet by asking this question? – sksallaj – 2016-03-14T21:30:54.910

6@sksallaj: If you have a look at the first answer, it was pretty short. It doesn't take very long to find the data if you think you may know where to look... – Jon Skeet – 2016-03-15T06:48:36.997

635@Gareth: Nope, but checking the details of Shanghai time zone transitions at that period was my first port of call. And I've been working on time zone transitions for Noda Time recently, so the possibility of ambiguity is pretty much at the forefront of my thoughts anyway... – Jon Skeet – 2011-07-27T08:35:08.330

23@Jon: Out of curiosity, why did they set their clocks back by such a "weird" interval? Anything like an hour would have seemed logical, but how come it was 5:52mins? – Johannes Rudolph – 2011-07-27T13:22:16.943

49

@Johannes: To make it a more globally normal time zone, I believe - the resulting offset is UTC+8. Paris did the same sort of thing in 1911 for example: http://www.timeanddate.com/worldclock/clockchange.html?n=195&year=1911

– Jon Skeet – 2011-07-27T13:25:02.663

32@Jon Do you happen to know if Java/.NET copes with September in 1752? I always used to love showing people cal 9 1752 on unix systems – Mr Moose – 2011-07-27T14:01:29.783

5@Mr Moose: I believe Joda Time will do different things depending on the calendar you pick, and Noda Time will when we implement cutover calendars. I don't know about the "native" Java / .NET classes. – Jon Skeet – 2011-07-27T14:04:01.113

12Does this mean that, somewhere in the Java date libraries, there's logic handling this very edge case (and every other bizarre edge case)? – Yahel – 2011-07-27T20:06:32.217

11@yahelc: Nope - there's just general time zone code, which handles all kinds of historical time zone transitions. – Jon Skeet – 2011-07-27T20:07:44.573

14

@yahelc, to elaborate: time zone changes are basically a small set of different possible parameterized events. The code implements this set of events, and a timezone database supplies a list of events and parameters for each timezone. By the way, this timezone database can be updated independently of the JRE itself: http://www.oracle.com/technetwork/java/javase/tzupdater-readme-136440.html

– Michael Borgwardt – 2011-07-27T22:05:52.673

29So why the heck was Shanghai 5 minutes out of wack in the first place? – Igby Largeman – 2011-07-27T22:16:14.667

24@Charles: Lots of places were had less conventional offsets back then. In some countries, different towns each had their own offset to be as close to geographically correct as possible. – Jon Skeet – 2011-07-27T22:27:45.400

354@Charles: back then, travellers knew to expect local time to be different everywhere (because it was). Additionally, watches were mechanical and drifted quickly, so people we used to adjusting them according to the local clocktower every couple of days anyway, even if they did not travel. So how were the tower clocks (which also drifted) set? Most easily by setting them to 12:00 when the sun reached its daily peak... which was different in every place not on the same longitude. This was the norm pretty much everywhere until railroad timetables required some sort of standardization. – Michael Borgwardt – 2011-07-28T12:58:39.240

464But then : how on Earth has this kind of knowledge survived the ages, so that nearly a century ago, it is implemented in software ? In 2011, anyone who mention timezone oddities to a non-software engineer is looked upon like a nerd. (And really, people expect all software to abstract it, and they don't give a damn if it's ambigous, when they say 'noon', we software engineer should deal with it). But to imagine someone in Shangai, in December 1927, thinking it would be relevant to note such a thing down, and that somehow this information was never lost, deleted, anything ... mind's blown. – phtrivier – 2011-07-30T16:28:45.707

This is just like my country: even the past is uncertain! – Alex Byrth – 2018-09-14T13:07:09.117

Thats a really nice answer! – Spara – 2018-11-13T13:47:04.767

1 497

You've encountered a local time discontinuity:

When local standard time was about to reach Sunday, 1. January 1928, 00:00:00 clocks were turned backward 0:05:52 hours to Saturday, 31. December 1927, 23:54:08 local standard time instead

This is not particularly strange and has happened pretty much everywhere at one time or another as timezones were switched or changed due to political or administrative actions.

Michael Borgwardt

Posted 2011-07-27T08:15:58.380

Reputation: 291 858

54

@Jason: For the bedtime reading, I'd suggest the (now) IANA timezone database (previously administered by a lovely guy named Olson, I think) would be a great resource: http://www.iana.org/time-zones. As far as I know, a majority of the open source world (thus mentioned libraries) use this as their primary source of timezone data.

– Sune Rasmussen – 2012-03-11T21:25:45.897

596

The moral of this strangeness is:

  • Use dates and times in UTC wherever possible.
  • If you can not display a date or time in UTC, always indicate the time-zone.
  • If you can not require an input date/time in UTC, require an explicitly indicated time-zone.

Raedwald

Posted 2011-07-27T08:15:58.380

Reputation: 25 516

62@Raedwald: Sure you would - What is the UTC time for 1927-12-31 23:54:08? (Ignoring, for the moment, that UTC didn't even exist in 1927). At some point this time and date are coming into your system, and you have to decide what to do with it. Telling the user they have to input time in UTC just moves the problem to the user, it doesn't eliminate it. – Nick Bastin – 2012-02-19T22:39:53.870

58I feel vindicated at the amount of activity on this thread, having been working on date/time refactoring of a large app for almost a year now. If you're doing something like calendaring, you can't "simply" store UTC, as the definitions of time zones in which it may be rendered will change over time. We store "user intent time" - the user's local time and their time zone - and UTC for searching and sorting, and whenever the IANA database is updated, we recalculate all the UTC times. – taiganaut – 2012-12-07T22:34:54.807

67Conversion/storage into UTC really wouldn't help for the problem described as you would encounter the discontinuity in the conversion to UTC. – unpythonic – 2011-07-30T04:28:55.413

18@Mark Mann: if your program uses UTC internally everywhere, converting to/from a local time-zone only in the UI, you would not care about such discontinuities. – Raedwald – 2011-08-26T11:50:42.060

330

When incrementing time you should convert back to UTC and then add or subtract. Use the local time only for display.

This way you will be able to walk through any periods where hours or minutes happen twice.

If you converted to UTC, add each second, and convert to local time for display. You would go through 11:54:08 p.m. LMT - 11:59:59 p.m. LMT and then 11:54:08 p.m. CST - 11:59:59 p.m. CST.

PatrickO

Posted 2011-07-27T08:15:58.380

Reputation: 3 309

277

Instead of converting each date, you use the following code

long difference = (sDt4.getTime() - sDt3.getTime()) / 1000;
System.out.println(difference);

And see the result is:

1

Rajshri

Posted 2011-07-27T08:15:58.380

Reputation: 3 375

67I'm afraid that's not the case. You can try my code in you system, it will output 1, because we have different locales. – Freewind – 2012-05-16T05:39:51.340

10That's only true because you have not specified the locale in the parser input. That's bad coding style and a huge design flaw in Java -- its inherent localization. Personally, I put "TZ=UTC LC_ALL=C" everywhere I use Java to avoid that. In addition you should avoid every localized version of an implementation unless you are directly interacting with a user and explicitly want it. Don't to ANY calculations including localizations, always use Locale.ROOT and UTC timezones unless absolutely necessary. – user1050755 – 2014-11-26T15:53:42.060

197

I'm sorry to say that, but the time discontinuity has moved a bit in

JDK 6 two years ago, and in JDK 7 just recently in update 25.

Lesson to learn: avoid non-UTC times at all costs, except, maybe, for display.

user1050755

Posted 2011-07-27T08:15:58.380

Reputation: 6 615

19This is incorrect. The discontinuity isn't a bug - it's just that a more recent version of TZDB has slightly different data. For example, on my machine with Java 8, if you change the code very slightly to use "1927-12-31 23:54:02" and "1927-12-31 23:54:03" you'll still see a discontinuity - but of 358 seconds now, instead of 353. Even more recent versions of TZDB have yet another difference - see my answer for details. There's no real bug here, just a design decision around how ambiguous date/time text values are parsed. – Jon Skeet – 2014-10-30T17:50:27.370

5The real problem is that programmers don't understand that conversion between local and universal time (in either direction) is not and cannot be 100% reliable. For old timestamps the data we have on what local time was is shaky at best. For future timestamps political actions can change what universal time a given local time maps to. For current and recent past timestamps you can have the problem that the process of updating the tz database and rolling out the changes can be slower than than the implementation schedule of the laws. – plugwash – 2017-03-14T16:06:29.547

175

As explained by others, there's a time discontinuity there. There are two possible timezone offsets for 1927-12-31 23:54:08 at Asia/Shanghai, but only one offset for 1927-12-31 23:54:07. So, depending on which offset is used, there's either a one second difference or a 5 minutes and 53 seconds difference.

This slight shift of offsets, instead of the usual one-hour daylight savings (summer time) we are used to, obscures the problem a bit.

Note that the 2013a update of the timezone database moved this discontinuity a few seconds earlier, but the effect would still be observable.

The new java.time package on Java 8 let use see this more clearly, and provide tools to handle it. Given:

DateTimeFormatterBuilder dtfb = new DateTimeFormatterBuilder();
dtfb.append(DateTimeFormatter.ISO_LOCAL_DATE);
dtfb.appendLiteral(' ');
dtfb.append(DateTimeFormatter.ISO_LOCAL_TIME);
DateTimeFormatter dtf = dtfb.toFormatter();
ZoneId shanghai = ZoneId.of("Asia/Shanghai");

String str3 = "1927-12-31 23:54:07";  
String str4 = "1927-12-31 23:54:08";  

ZonedDateTime zdt3 = LocalDateTime.parse(str3, dtf).atZone(shanghai);
ZonedDateTime zdt4 = LocalDateTime.parse(str4, dtf).atZone(shanghai);

Duration durationAtEarlierOffset = Duration.between(zdt3.withEarlierOffsetAtOverlap(), zdt4.withEarlierOffsetAtOverlap());

Duration durationAtLaterOffset = Duration.between(zdt3.withLaterOffsetAtOverlap(), zdt4.withLaterOffsetAtOverlap());

Then durationAtEarlierOffset will be one second, while durationAtLaterOffset will be five minutes and 53 seconds.

Also, these two offsets are the same:

// Both have offsets +08:05:52
ZoneOffset zo3Earlier = zdt3.withEarlierOffsetAtOverlap().getOffset();
ZoneOffset zo3Later = zdt3.withLaterOffsetAtOverlap().getOffset();

But these two are different:

// +08:05:52
ZoneOffset zo4Earlier = zdt4.withEarlierOffsetAtOverlap().getOffset();

// +08:00
ZoneOffset zo4Later = zdt4.withLaterOffsetAtOverlap().getOffset();

You can see the same problem comparing 1927-12-31 23:59:59 with 1928-01-01 00:00:00, though, in this case, it is the earlier offset that produce the longer divergence, and it is the earlier date that has two possible offsets.

Another way to approach this is to check whether there's a transition going on. We can do this like this:

// Null
ZoneOffsetTransition zot3 = shanghai.getRules().getTransition(ld3.toLocalDateTime);

// An overlap transition
ZoneOffsetTransition zot4 = shanghai.getRules().getTransition(ld3.toLocalDateTime);

You can check whether the transition is an overlap - in which case there's more than one valid offset for that date/time - or a gap - in which case that date/time is not valid for that zone id - by using the isOverlap() and isGap() methods on zot4.

I hope this helps people handle this sort of issue once Java 8 becomes widely available, or to those using Java 7 who adopt the JSR 310 backport.

Daniel C. Sobral

Posted 2011-07-27T08:15:58.380

Reputation: 252 564

1

Hi Daniel, I have run your piece of code but it is not giving output as expected. like durationAtEarlierOffset and durationAtLaterOffset both are 1 second only and also zot3 and zot4 both are null. I have set just copied and run this code on my machine. is there anything which needs to be done here. Let me know if you want to see a piece of code. Here is code http://www.tutorialspoint.com/compile_java8_online.php?PID=0Bw_CjBb95KQMUTIyLU1tQ2x3bDQ can you let me know what going on here.

– vineeshchauhan – 2017-06-15T10:30:00.783

1@vineeshchauhan It depends on Java's version, because this has changed in tzdata, and different versions of JDK bundle different versions of tzdata. On my own installed Java, the times are 1900-12-31 23:54:16 and 1900-12-31 23:54:17, but that doesn't work on the site you shared, so they are using a different Java version than I. – Daniel C. Sobral – 2017-07-21T00:38:05.433

145

IMHO the pervasive, implicit localization in Java is its single largest design flaw. It may be intended for user interfaces, but frankly, who really uses Java for user interfaces today except for some IDEs where you can basically ignore localization because programmers aren't exactly the target audience for it. You can fix it (especially on Linux servers) by:

  • export LC_ALL=C TZ=UTC
  • set your system clock to UTC
  • never use localized implementations unless absolutely necessary (ie for display only)

To the Java Community Process members I recommend:

  • make localized methods not the default, but require the user to explicitly request localization.
  • use UTF-8/UTC as the FIXED default instead because that's simply the default today. There is no reason to do something else, except if you want to produce threads like this.

I mean, come on, aren't global static variables an anti-OO pattern? Nothing else is those pervasive defaults given by some rudimentary environment variables.......

user1050755

Posted 2011-07-27T08:15:58.380

Reputation: 6 615