Magic behind FreeAndNil

Delphi 10.4 Sydney brings one small but rather significant change. A change in the signature of the FreeAndNil procedure from

procedure FreeAndNil(var Obj); inline;

to

procedure FreeAndNil(const [ref] Obj: TObject); inline;

Wait, const means the variable is passed as a constant and cannot be changed from within the procedure. But the whole point of FreeAndNil is to change the passed object variable to nil. What kind of sorcery is this?

Let's start from the beginning and the old FreeAndNil procedure, its purpose and implementation.
The purpose of FreeAndNil is to simplify the following code sequence:

Foo.Free;
  Foo := nil;

While the actual implementation of FreeAndNil is slightly different, and it will first nil the reference and then Free the object instance (temporarily preserved in a local variable), that slight difference is not relevant for discussing the signature of FreeAndNil.

procedure FreeAndNil(var Obj);
var
  Temp: TObject;
begin
  Temp := TObject(Obj);
  Pointer(Obj) := nil;
  Temp.Free;
end;

In order to apply a change (nil assignment) to a variable (reference) outside the procedure, Obj must be passed as a var parameter. But, if we use a typed var parameter and write the following signature

procedure FreeAndNil(var Obj: TObject);

we could only pass variables declared as TObject or we would get the compiler error

E2033 Types of actual and formal var parameters must be identical

This rule is enforced by the compiler for good reason. Without it you could have a "pass the dog, get the cat" situation, which would be bad if the compiler would allow that on all var parameters - we would lose type safety in all code where the var paramater is used. To avoid the above problem and be able to use FreeAndNil with any kind of object reference, Obj must be an untyped parameter.

And that brings us to the main problem with the original FreeAndNil signature. It is not type safe. The compiler will merrily let us pass in just about anything, like interface references, records, plain pointers... and making such mistakes will cause random errors and memory corruption at runtime, depending on the actual content of the passed variable.

Since the first FreeAndNil implementation was written, Delphi got an additional parameter decorator, [ref], which forces the compiler to pass a const parameter as a reference. Without it, the compiler can optimize passing of const parameter depending on the size of the parameter. Why it was introduced and what problems it originally solves, is another story (just like the cats and dogs one), for this one what matters is what it does.

So let's see what exactly the real magic is behind the new FreeAndNil signature and how the new procedure is actually implemented.

procedure FreeAndNil(const [ref] Obj: TObject);
var
  Temp: TObject;
begin
  Temp := Obj;
  TObject(Pointer(@Obj)^) := nil;
  Temp.Free;
end;

Instead of using var, the new signature uses const [ref]. const gives us type safety and allows passing any TObject reference or its descendants. This is exactly what we need, we want object variables and nothing but object variables. [ref] enforces that the Obj variable itself will be passed as a reference (pointer). Here it is important to make a distinction between the TObject type that is the reference type itself, and the variable that stores the reference to the actual object instance.

Without using [ref], the contents of the Obj variable (a pointer to the object) would be copied to the stack - we would lose the connection to the original variable we are passing to the procedure, and we would not be able to nil it.

Basically, the const [ref] combination will generate the same code for passing the Obj parameter as it would if we are using a var parameter. This is a crucial requirement for niling the original variable.
But, here const throws us a curveball. It basically tells the compiler to throw an error if we attempt to change the Obj parameter inside the procedure.

procedure FreeAndNil(const [ref] Obj: TObject);
var
  Temp: TObject;
begin
  Temp := Obj;
  Obj := nil; // E2064 Left side cannot be assigned to
  Temp.Free;
end;

The above problem could be easily solved using inline assembler. We could do anything needed and the compiler would (could) not complain. The problem is that we need FreeAndNil on all platforms, even ones where Delphi does not support inline assembler.

Fortunately, Delphi itself can go pretty low level and by using pointers and a little type casting we can write

TObject(Pointer(@Obj)^) := nil;

which roughly translates to "get me the address of the variable Obj, cast it to Pointer, and then cast the contents of that Pointer to TObject and nil it".

The real magic is that when you do all the pointer typecasting, the compiler no longer recognizes that you have actually changed the contents of the Obj variable you were not supposed to change because it is passed as const.

Problem solved - we have type safe FreeAndNil!!!

One of the potential problems with any kind of hacky solutions that rely on implementation details is that implementation details can change and such changes can break the code.

Knowing which implementation details can possibly change and which don't often requires deeper knowledge about how and why, even to the "only for compiler guys" level of understanding. Not very practical from the perspective of a Delphi developer who does not need to know such things.

Some of the tricks used in the new FreeAndNil implementation are documented and their details will not change. But, there is no guarantee that at some point in time, the compiler will not get some improvement that will accidentally or purposefully break the "confuse the compiler with typecasting const parameter" code.

Don't fret! The FreeAndNil hack will not be broken in the future, and you can use the above tricks in your own code, too. To paraphrase what has been said about potential future changes in that area: "There is always a possibility that the tricks used in FreeAndNil will stop working, but since we used them in FreeAndNil, we will make sure we don't break them".

Disclaimer: It is almost impossible to predict future compiler changes, sometimes, on rare occasions, some widely used features and implementation details must change in order to implement some more important features.