Robbie Hanson:

You cannot store a plain-old NSString in Core Data. It must be wrapped inside a NSManagedObject. Why? Because of the mandate above. An NSString is immutable. And Core Data wants to update the object you have in your hand, at the moment the context merges changes from another context. And it can’t change an immutable object. So it must wrap it inside something else, and change that.

[…]

This also has implications on threading. Why is it not safe to pass a NSManagedObject to background threads? Because that darn NSManagedObjectContext may mutate the thing at any time. So now threading becomes a nightmare. The fact is, the rules for NSManagedObject don’t follow the same rules you have for all other objects. Developers understand mutable vs immutable. Especially in objective-c where there are often 2 versions of a class, one mutable and one immutable. And developers understand the implications concerning threading: “Don’t be mutating something on one thread if it’s being used on another.” This is something we’ve lived with for a long time, and we’re quite adept at solving these problems. But NSManagedObject presents a whole new set of problems. We can’t just wrap one of these things in a lock, or only access it through a serial queue. Because NSManagedObjectContext isn’t going to follow our rules. It won’t play nicely, and so we’re forced to play entirely by its rules.

In contrast:

YapDatabase doesn’t use NSManagedObject, or any similar kind of silliness. You can store plain-old NSObjects. This includes your own NSObject subclasses, as well as the usual suspects such as NSString, NSNumber, UIColor, UIImage, etc. As long as the object can somehow be serialized & deserialized you’re good to go. This could be via NSCoding, or any other way you want such as JSON serialization. It’s completely configurable.

[…]

YapDatabase will never mutate any of your objects. Ever. This gives you complete control over your objects, and how & when you access and mutate them. You know, the way you’re used to dealing with objects.

YapDatabase looks really interesting. I skimmed the wiki to try to figure out the general architecture. It’s based on SQLite, which it uses as a collection-key-value store. In contrast to Core Data, it’s quasi-schemaless, although it does have some support for relationships.

It looks like there are four different mechanisms for selecting/filtering objects:

• By primary (string) key.
• Using a secondary SQLite index, which you explicitly set up and query using SQL fragments.
• Using Objective-C code on full in-memory objects that have been deserialized.
• Using SQLite’s FTS module.

Benedict Cohen:

parentContext was introduced alongside a new concurrency model. To use parentContext both the parent and child contexts must adopt the new concurrency model. But the problem addressed by parentContext is not concurrency. Concurrency is just a problem, albeit a significant one, that needed to be solved to for parentContext to be implemented. The intent of parentContext is to improve the [atomicity] of changes. parentContext allows changes to be batch up and committed en masse. This has always been possible by using multiple NSManagedObjectContext, but parentContext allows for improved granularity of the batching.

[…]

parentContext does provides features that simplify a handful of use cases. Unfortunately the short comings of parentContext mean that it can not be adopted piecemeal. The top of the Core Data stack are managed objects. A good model will provide an interface that works at a high level of abstraction. Creating such an interface requires encapsulating implementation detail. The way that Core Data is designed means that the natural place for this code is in managed object subclasses. Because parentContext affects the behaviour of managed objects adopting it makes it difficult to write managed object subclasses without knowing the context hierarchy in which they’ll be used. Proceed with extreme caution!

Marcus Zarra:

Unlike a relationship, there is no way to pre-fetch a fetched property. Therefore, if you are going to fetch a large number of entities and then desire to access the fetched property for those properties, they are going to be fetched individually. This will drastically impact performance.

Fetched Properties are only fetched once per context without a reset. This means that if you add other objects that would qualify for the fetched property after the property has been fetched then they won’t be included if you call the fetched property again. To reset the fetched property requires a call to -refreshObject:mergeChanges:.

Brent Simmons:

Model objects live on the main thread. This makes it easy to use VSNote, VSTag, and so on in view controllers and in syncing.

There is one exception: you can create a “detached” copy of a model object to use with API calls. A detached model object exists on one thread of execution only, is short-lived, and is disconnected from the database. Detached objects aren’t a factor when it comes to concurrency.

When a model object is added, changed, or deleted, updates to the database are placed in a background serial queue.

Update (2014-03-07): Jesper:

Of Apple’s fixes’ own admission, Core Data sync didn’t work because it was a black box with no ability to debug it. It would be unfair to zing Core Data at large with that epithet. But if it’s something that seems true about Apple’s frameworks, love them mostly as I do, it’s that they’re constructed as if to impress on their user how privileged they should feel because of the difficulty of the bar that they set to solve the problem at, and the complexity of implementation they have used to convincingly solve the problem.

[…]

Basic features are still painful for people that have been successful Cocoa coders for ten years. They’re not sufficiently saved by the ripening of frameworks as much as by their own accumulated ingenuity. Cocoa is still being developed, features are added, but rarely does something hard get easier.

Brent Simmons:

The second reason has to do with my enduring love of plain-ol’ Cocoa. I like regular Cocoa objects. I like being able to implement NSCoding, override isEqual: and hash, and design objects that can be created with a simple init (when possible and sensible). I especially like being able to do those things with model objects. (Which totally makes sense.)

Pablo Bendersky:

When you use a Migration Manager, Core Data will create a new database for you, and start copying the entities one by one from the old DB to the new one.

As we are using journal_mode = WAL, there’s an additional file besides DB.sqlite called DB.sqlite-wal.

From what I can tell, the problem seems to be that Core Data creates a temporary DB, inserts everything there, and when it renames it to the original name, the -wal file is kept as a leftover from the old version. The problem is that you end up with an inconsistent DB.

A different part of Core Data is aware of the multiple files, though:

To safely back up and restore a Core Data SQLite store, you can do the following:

• Use the following method of NSPersistentStoreCoordinator class, rather than file system APIs, to back up and restore the Core Data store:

  - (NSPersistentStore *)migratePersistentStore:(NSPersistentStore *)store toURL:(NSURL *)URL options:(NSDictionary *)options withType:(NSString *)storeType error:(NSError **)error

  Note that this is the option we recommend.

• Change to rollback journaling mode when adding the store to a persistent store coordinator if you have to copy the store file.

Findings from Charles Parnot et. al. looks like a neat app:

When doing science and running experiments, it is crucial to keep track of what one is doing, to be able to later reproduce the results, assemble and publish them. This is what lab notebooks are for. There is something great about paper, and the freedom and flexibility it affords. But in 2014, paper starts to show its limits in other areas where computers have taken over: storing results, analysing data, searching, replicating, sharing, preserving, and more.

Findings ambition is simple: make your computer a better tool than paper to run your experiments and keep your lab records.

Findings stores experiments, which can be organized into projects, and protocols, which cut across experiments:

Protocols are the primary building blocks of your experiments. You can drag them into the calendar view of an experiment and combine them in any way you need. Once integrated into an experiment, a copy of your protocol is made, so you can modify it just for that one use, and leave the original untouched.

It’s currently Mac-only, but they intend to add support for iOS and syncing. The storage layer, the open-source PARStore, is designed for cloud-agnostic syncing, e.g. Dropbox or iCloud document storage.

PARStore has an interesting design:

1. It’s a key-value store.
2. There’s one copy of the store for each device, and each device has all of the copies. The device opens its own copy read-write and the other copies read-only.
3. Each copy is a log of timestamped changes, i.e. “Set key to value.” It only ever adds to the log, so there can be many entries for a given key.
4. For each key, the entry with the latest timestamp is the truth. Presumably you could rewind to an earlier version of the document by searching for timestamps before a given date.
5. Periodically, each device reads the other device’s logs and incorporates any changes with more recent timestamps into its own log.
6. Each log is implemented as a Core Data SQLite database.
7. There can also be attached files (blobs), which are stored outside of the database.

This seems like an elegant solution for synchronizing modest amounts of data, provided that it’s suited to key-value rather than row-column storage. The crux of it is that, normally, multiple devices cannot simultaneously open the same database file because SQLite’s locks don’t work across Dropbox. Or, rather, you can do it, but you’ll probably corrupt the database. PARStore gets around this by allowing multiple readers per file but only one writer. I’m not convinced that this will work 100% of the time, though:

1. There doesn’t seem to be a mechanism to prevent Core Data from opening a database that Dropbox is in the process of writing to (e.g. updating it for changes made on other devices). It’s only opening it as read-only, so this shouldn’t corrupt the file, but it’s probably undefined what happens when it tries to read from the file.
2. Even if the database file is fully written, there’s no telling whether the adjacent -wal and -shm files match it to form a consistent whole.
3. It’s not entirely clear to me how SQLite handles read-only databases in WAL mode. The documentation implies that write access is needed. If it’s writing to the -shm file even in read-only mode, that might cause problems for the device that’s opened the database read-write.

That said, with a good Internet connection and relatively small files, I doubt that there would be many problems in practice.

Update (2014-05-30): Charles Parnot:

Findings has only been out for 8 days, and I am really proud of the launch, impressed by the response and excited about all the work that’s ahead. But before marching into the future, I thought I should look back into the past. While the core functionality of the app has remained the same, it is quite amazing to see how much of the look and the design of the app has changed over the years… I am a big fan of ‘making of’ posts on apps. I wish there were more of these, so here is one for Findings!

Tom Harrington:

CloudKit makes a refreshing change from iCloud Core Data in that there’s a lot less magic going on in the framework. Using iCloud to sync Core Data is very slick, in that you can essentially treat changes from the cloud as if they had been made on a different thread. Changes get saved, you get notified, and you merge those changes and update the app state. Your code never concerns itself directly with the cloud aspects of its data. That’s when it works, of course. It does appear that iCloud with Core Data works much better than it did when I last tangled with it, but it’s still a magic box that abstracts away all of the work of dealing with the cloud.

In contrast CloudKit is not actually a sync mechanism. Instead it’s a transfer mechanism, where your app must explicitly initiate all data transfers. Starting with the CKDatabase class, your app gets local references to cloud-based databases. CloudKit’s API deals with that database as a specifically cloud-based entity. When new changes are available, you need to fetch CKRecord instances and convert them into something that fits your local model. When you have new outgoing changes, you need to create or update the appropriate CKRecord, again converting data as needed.

[…]

As a result, using CloudKit will likely mean more code in your app than using iCloud with Core Data. On the other hand, CloudKit’s more direct and less magical approach to dealing with cloud-based data should be more reliable.

Alexander Edge:

I have benchmarked the difference between migration using binary data with external storage and using just a unique identifier. The sample project is on GitHub.

The test creates 1000 objects, each with a unique identifier and (optionally) a video file of 165KB as binary data. By adding a data model version with a new attribute, lightweight migration is performed on the next application launch.

When using binary data with external storage, migration took 37.5 seconds.

When not using binary data with external storage, migration took just 0.6 seconds.

It’s clear to see that storing large files in the file system with a naming scheme is the way to go. I have already begun working on an on-disk cache instead of using binary attributes.

External BLOB storage is important because storing BLOBs as linked lists in the database can be inefficient. When Core Data added -setAllowsExternalBinaryDataStorage:, this sounded like a good solution to the problem, but various problems were reported in early versions. It sounds like there are still issues, which is unfortunate because this sort of feature (commonly needed, tricky to do well) should be a selling point for a more full-featured framework like Core Data.

Christian Kienle:

In a little while I will remove Core Data Editor from the Mac App store and offer Core Data Editor as a free download from my website. One of the main reasons for doing so is my health. I can maintain a project like Core Data Editor but I cannot add significant features on my own anymore.

GitHub:

Core Data Editor lets you easily view, edit and analyze applications‘ data. Core Data Editor is compatible with Mac and iOS applications and supports XML, SQLite and binary stores, visualizes all relationships and is able to edit the data and generate Objective-C code for the data model.

Geppy Parziale:

First of all, Apple introduced a new method executeRequest:error: in the NSManagedObjectContext class. It is very similar to the executeFetchRequest:error: method that you can use to perform a fetch request to populate the Managed Object Context. Its first argument is a NSBatchUpdateRequest. This is a new class, subclass of the NSPersistentStoreRequest recently introduced in iOS 7, and provides very similar functionalities to its sibling NSFetchRequest. The batch request is composed of an entity (the entity containing the property or properties you want to update) and a predicate to define a subset of data you want to update. Eventually, you can also define a dictionary containing the properties you want to update and their new values.

Once created, the NSBatchUpdateRequest is passed to the executeRequest:error: method. After its execution, this method returns an NSBatchUpdateResult object (subclass of NSPersistentStoreResult), the result property of which contains the batch updates result value(s). You can define the type of results you want from the executeRequest:error:, when you define the NSBatchUpdateRequest. You choose among three types of results:

[…]

The legacy Fetch-Update-Save takes 7.2s to run on the iPhone 5s, while the new batch updates run in only 0.81 s.

Impressive is also the memory usage. As expected, since the batch updates run directly on the Persistent Store, the memory usage is incredibly lower than the old approach.

I’ve been wanting something like this for a long time. (Note that it doesn’t handle deletion.)

Adam Ernst and Ari Grant gave a presentation at @Scale 2014 about why Facebook switched away from Core Data and MVC to immutable models and their own layout engine that separates the layout from the process of creating the view hierarchy from recycled immutable views.

James Dempsey and the Breakpoints (iTunes):

Backtrace steps through fourteen years of Mac and iOS development tunes, taking you on a musical journey into the biggest album release in iOS and Mac programming history.

From the driving beat of Goto Fail to the memory management oldie Hold Me, Use Me, Release Me every song is here. From crowd favorites to deep cuts, each track melds music with humor-filled tech lyrics, welcoming you to a sonic wonderland of geektastic amusement.

Update (2014-11-24): Dempsey is posting the lyrics.

Nat!:

The project started out on iOS 5, which was quickly succeeded by iOS 6. I would have been extremely surprised at the beginning, if someone had told me, that at the time of iOS 8s release our app still wouldn’t be done yet. But here is a recollection of all my faults: why it took way too long.

[…]

I bought AppCode solely to run “Inspect Code…”. The results returned are quite a bit more helpful than what Xcode Analyzer returns.

[…]

I probably wrote a hundred little apps, that tested out some feature, or started coding a subview with it. When the code was complete I moved it into the main app, deleted the original files and then symlinked the files from the main app in the test app. This way, I could go back to the test app to tweak something, when it didn’t work out in the main app. Needless to say being able to focus on just a small piece of code in a controlled environment is much more convenient.

[…]

This unfortunately means, that I am almost invariably are going to hit a brick wall at some point in time. For example, I spent way, way more time dicking around with UIScrollView than I eventually needed to code my own custom UIScrollView. The opacity of the iOS libraries means, that I always have to guess, how it’s really implemented, guess how it could break in the next iOS version and also guess beforehand, if everything is exposed like I will eventually need it.

[…]

Subclassing CoreData classes or overriding CoreData accessors is a path to misery, where I am unfortunately still traveling on. I am not 100% sure, but I would probably have been better off, either just going sqlite-direct or to use a stripped down MulleEOF for Dienstag.

[…]

It was interesting, because “naive code” only suffered a factor 2 ARC penalty, whereas “clever code” suffered a factor 10 ARC penalty. So ARC seems to be a great programmer equalizer in that respect. I didn’t investigate other “patterns”, but I also continued not using ARC. Less magic, less pain.

John Brayton (via John Gordon):

The most significant improvement in the CloudPull 2.6 update is Yosemite compatibility. I submitted this update to the Mac App Store on October 1. Apple rejected it on October 15. I spent time from October 15 through October 27 escalating the issue — first with the App Review Team, then with the App Review Board, and finally with Apple Developer Technical Support. Meanwhile, Yosemite was released on October 16 and many customers have already migrated to the version available on my web site.

It is possible to work around Apple’s reason for rejecting the app. Instead, I have decided to distribute CloudPull exclusively through my web site. That will allow me to provide updates more quickly, and to focus on serving customers rather than dealing with the Mac App Store.

The reason for the rejection was an OS bug that causes Core Data to open a compiled model file in read-write mode when it only needs read access. The .mom file is probably inside the app’s bundle, which is not a location that apps are supposed to be writing to. Lots of developers ran into problems like this when the Mac App Store was new. Apple seems to be using DTrace to audit what submitted apps are doing, but it still hasn’t audited its own APIs to make sure that they play by the rules.

It sounds like Brayton did everything by the book, but one month after submission his app was still in limbo when he pulled the plug on the Mac App Store version. This is the sort of Max Q issue that Rich Siegel was talking about last month. The Mac App Store is supposed to make it easier for customers to keep their apps up-to-date. In this case, it instead impeded the process of delivering a Yosemite compatibility update.

Tom Harrington notes that several developers (including James Dempsey and Rich Siegel) have encountered a bug with Core Data’s -[NSPersistentStoreCoordinator migratePersistentStore:toURL:options:withType:error:] method. This is supposed to copy the data store from one location to another, e.g. for making a backup. The frameworks may also call the method on your behalf, e.g. if the user does a Save As. The bug is that, on Yosemite and iOS 8, the method discards many-to-many relationships. This is actually worse than corrupting the file because the new file will still be readable, so it may not be obvious that it’s missing some of the data. The bug is reputed to affect all store types.

(DropDMG and EagleFiler use Core Data but not many-to-many relationships, so I don’t think they are affected.)

Update (2014-11-24): Harrington has filed a bug and posted a sample project.

Update (2015-03-10): This is fixed in Mac OS X 10.10.2 and iOS 8.2.

Dan Counsell:

One of the biggest challenges we faced last year with Ember was sync. Once you start using Ember, it’s incredibly easy to build up a substantial library - and customers upgrading from LittleSnapper have sometimes many thousands more images imported! Multi-gigabyte libraries are not uncommon, and combined with (in particular) iCloud customers have found sync to be problematic. It especially doesn’t help that iCloud is a very hard-to-inspect black box, and there’s no progress shown to you as to the uploaded (or not) state of your library. It’s tough to say it, as iCloud’s robustness has continued to improve each year, but right now iCloud isn‘t a good fit for apps like Ember, and our future product direction will reflect that.

Last year they were optimistic that iCloud Core Data would improve. Now—three plus years after iCloud Core Data shipped—it sounds like they are giving up on it.

Tim Oliver (tweet):

Tim Oliver, creator of the iComics app, gave a talk featuring Realm in his project at the Perth iOS Meetup. His talk covered gave a quick background of his transition to and experiences using Realm in place of Core Data, a switch he made in just one evening.

[…]

When I was working on iComics, I’d have problems when a background thread would import a comic, tell the main thread the comic imported, then send an object with entirely empty fields. When I tested installing a new version of the app over an older version, I found that Core Data’s auto-migration was corrupting half the database and making the app very unstable. A manual Core Data migration would have required a ridiculous amount of code, and I decided to switch to Realm.

The Mac OS X 10.10.2 update fixes the Thunderstrike exploit, among other security bugs. (Good luck if you have a Thunderbolt-equipped Mac but aren’t running Yosemite.)

It claims to fix Wi-Fi issues, but Macworld and MacRumors both report that Wi-Fi problems persist.

It does fix the nasty Core Data relationships data loss bug, though this is not yet fixed in iOS.

Launchpad’s sparkle for new applications has been removed.

Lloyd Chambers is still seeing problems with dual displays. I’m still seeing problems with the login screen, Mail, etc.

Update (2015-02-03): Edward Marczak:

To call [Thunderstrike], “fixed,” is absolute fantasy. A good chunk of the boot process would need to change to fix this.

Kirk McElhearn:

So, there are a couple of morals to this story. First, don’t trust iCloud. I don’t blame Realmac; it’s probably not their fault. I find it interesting that the data loss occurred overnight; I may have looked at my Clear list on my iPad yesterday evening, but I don’t recall doing so. If I had, I would have spotted that a list was missing. So my guess is that the data loss is due to iCloud itself.

Second, don’t use an app that doesn’t offer backups. I looked in my local folders and couldn’t find any readable files that I would have been able to recover from Time Machine backups. There’s a locally-stored .sqlite file, but it doesn’t contain the same data.

I won’t be trusting any important data to iCloud any more. I am slightly concerned about my contacts; I’ve had issues with them in the past. I know there are local and Time Machine backups that I could restore if I ever need to. But for the rest, iCloud is simply too precarious.

He used iExplorer to get the Core Data SQLite database out of his local iTunes backup.

Matt Drance:

1) Create NSManagedObjectContext

2) Fetch

2) Pass results to a method

I’ve seen ARC kill the MOC before line 3.

With at-autoreleasepool, you could optimize given advanced knowledge. Now you need advanced knowledge to not die.

The problem here is that Core Data expects that you will keep the managed object context alive as long as you are using any of its objects, but the objects do not retain their context:

Managed objects know what managed object context they’re associated with, and managed object contexts know what managed objects they contain. By default, though, the references between a managed object and its context are weak. This means that in general you cannot rely on a context to ensure the longevity of a managed object instance, and you cannot rely on the existence of a managed object to ensure the longevity of a context.

ARC will try to release the context after the last line of code that references it, even if this is before the end of the scope. (This is different from other automated reference counting systems, such as Python’s.) It doesn’t know that later code in that scope may use managed objects that assume their context is still alive. This was also a problem with Objective-C garbage collection; Core Data used weak references even though the collector could have cleaned up cyclic strong ones.

Some potential solutions:

• Make sure the context is owned by some higher level object so that it stays alive. This may not make sense for temporary contexts, though.

• With garbage collection, a workaround was to send an innocuous message such as [context class] at the end of the scope so as to keep the object alive until that point. This still works under ARC.

• You can annotate the context’s variable with the objc_precise_lifetime attribute to tell ARC that you want it kept alive until the end of the scope. As of Mac OS X 10.8, there is also the Foundation-level NS_VALID_UNTIL_END_OF_SCOPE macro for doing this:

  Marks local variables of type id or pointer-to-ObjC-object-type so that values stored into those local variable are not aggressively released by the compiler during optimization. Instead, the values are held until either the variable is assigned to again, or the end of the scope of the local variable (such as in a compound statement or a method definition).

All of this makes me sad because something this basic should not be so error-prone, and it should not be getting worse with time. As Drance notes, all of this just worked with manual reference counting. An autoreleased context would automatically stay alive as long as it was needed. You could fine-tune object lifetimes for increased performance; now you have to do so to avoid crashes.

Arguably, this is a design flaw in Core Data rather than ARC. Why don’t objects retain their context?

As far as I can tell, the situation is actually worse with Swift. It uses ARC but doesn’t seem to have the equivalent of NS_VALID_UNTIL_END_OF_SCOPE.