It's hard to match nested parenthesis pairs using plain regexp:s. If you use code instead, you can find the first and use
forward-sexp to find the closing. The next step is to manufacture a match-data, so that you can match the brackets using subexpression font-lock highlights.
The following example is from obj-font-lock, a package the highlight Objective-C function calls:
(defun objc-font-lock-match-bracket (lim)
"Search for an Objective-C method call.
Return non-nil if one is found. The point will be placed after
the start brace. The match data will be filled with the following:
0 -- The entire method call
1 -- The start brace
2 -- The object expression
3 -- The end brace.
Note that this function is intended to be used by font-lock, and
assumes that comments and string have been fontified."
(let ((ok nil)
;; Look for "[":s, but skip those that are used for array constructs.
(setq ok (re-search-forward "\\[" lim t))
;; Continue searching, if this bracket isn't suitable.
;; When the `[' doesn't match a `]', skip it.
;; Without this, code after the method call would
;; be fontfied, typically up the the next closing
(goto-char (match-beginning 0))
(not (eq (char-before) ?\] )))
;; Skip array subscripts.
;; Record the start end end location of the first
;; complex expression, in case this wasn't an array
(setq beg-of-expr (point))
(setq end-of-expr (point))
(eq (following-char) ?\]))))))
(let* ((beg (- (point) 1))
;; This synthesized match data will be in place for the
;; highlights of the keyword as well as when the pre- and
;; post-match-forms of the anchored rule are executed.
;; Match 0: Full range
;; Match 1: [
beg (+ beg 1)
;; Match 2: expr
;; Match 3: ]
(- end 1) end))))